The apolipoprotein E (APOE) gene has been consistently shown to modulate the risk of Alzheimer's disease (AD). Here, using an AD and normal aging dataset primarily consisting of three AD multi-center studies (n 1,781), we compared the effect of APOE and amyloid-β (Aβ) on baseline hippocampal volumes in AD patients, mild cognitive impairment (MCI) subjects, and healthy controls. A large sample of healthy adolescents (n 1,387) was also used to compared hippocampal volumes between APOE groups. Subjects had undergone a magnetic resonance imaging (MRI) scan and APOE genotyping. Hippocampal volumes were processed using FreeSurfer. In the AD and normal aging dataset, hippocampal comparisons were performed in each APOE group and in ϵ4 carriers with positron emission tomography Aβ who were dichotomized (Aβ/Aβ-) using previous cut-offs. We found a linear reduction in hippocampal volumes with ϵ4 carriers possessing the smallest volumes, ϵ3 carriers possessing intermediate volumes, and ϵ2 carriers possessing the largest volumes. Moreover, AD and MCI ϵ4 carriers possessed the smallest hippocampal volumes and control ϵ2 carriers possessed the largest hippocampal volumes. Subjects with both APOE ϵ4 and Aβ had the lowest hippocampal volumes when compared to Aβ- ϵ4 carriers, suggesting a synergistic relationship between APOE ϵ4 and Aβ. However, we found no hippocampal volume differences between APOE groups in healthy 14-year-old adolescents. Our findings suggest that the strongest neuroanatomic effect of APOE ϵ4 on the hippocampus is observed in AD and groups most at risk of developing the disease, whereas hippocampi of old and young healthy individuals remain unaffected.