Visual processing abnormalities in schizophrenia (SZ) are poorly understood, yet predict functional outcomes in the disorder. Bipolar disorder (BD) may involve similar visual processing deficits. Converging evidence suggests that visual processing may be relatively normal at early stages of visual processing such as early visual cortex (EVC), but that processing abnormalities may become more pronounced by mid-level visual areas such as lateral occipital cortex (LO). However, little is known about the connectivity of the visual system in SZ and BD. If the flow of information to, from, or within the visual system is disrupted by reduced connectivity, this could help to explain perceptual deficits. In the present study, we performed a targeted analysis of the structural and functional connectivity of the visual system using graph-theoretic metrics in a sample of 48 SZ, 46 BD, and 47 control participants. Specifically, we calculated parallel measures of local efficiency for EVC and LO from both diffusion weighted imaging data (structural) and resting-state (functional) imaging data. We found no structural connectivity differences between the groups. However, there was a significant group difference in functional connectivity and a significant group-by-region interaction driven by reduced LO connectivity in SZ relative to HC, whereas BD was approximately intermediate to the other 2 groups. We replicated this pattern of results using a different brain atlas. These findings support and extend theoretical models of perceptual dysfunction in SZ, providing a framework for further investigation of visual deficits linked to functional outcomes in SZ and related disorders.