Theoretical background: The Apolipoprotein E (APOE) ε4 genotype is known to be one of the strongest single-gene predictors for Alzheimer disease, which is characterized by widespread brain structural degeneration progressing along with cognitive impairment. The ε4 allele status has been associated with brain structural alterations and lower cognitive ability in non-demented subjects. However, it remains unclear to what extent the visuospatial cognitive domain is affected, from what age onward changes are detectable and if alterations may interact with cognitive deficits in major depressive disorder (MDD). The current work investigated the effect of APOE ε4 homozygosity on visuospatial working memory (vWM) capacity, and on hippocampal morphometry. Furthermore, potential moderating roles of age and MDD were assessed. Methods: A sample of n = 31 homozygous ε4 carriers was contrasted with n = 31 non-ε4 carriers in a cross-sectional design. The sample consisted of non-demented, young to mid-age participants (mean age = 34.47; SD = 13.48; 51.6% female). Among them were n = 12 homozygous ε4 carriers and n = 12 non-ε4 carriers suffering from MDD (39%). VWM was assessed using the Corsi block-tapping task. Region of interest analyses of hippocampal gray matter density and volume were conducted using voxel-based morphometry (CAT12), and Freesurfer, respectively. Results: Homozygous ε4 carriers showed significantly lower Corsi span capacity than non-ε4 carriers did, and Corsi span capacity was associated with higher gray matter density of the hippocampus. APOE group differences in hippocampal volume could be detected but were no longer present when controlling for total intracranial volume. Hippocampal gray matter density did not differ between APOE groups. We did not find any interaction effects of age and MDD diagnosis on hippocampal morphometry. Conclusion: Our results point toward a negative association of homozygous ε4 allele status with vWM capacity already during mid-adulthood, which emerges independently of MDD diagnosis and age. APOE genotype seems to be associated with global brain structural rather than hippocampus specific alterations in young- to mid-age participants.