The N400, a negative event-related potential (ERP) peaking approximately 400 ms after stimulus onset, is known to reflect the processing of semantic information. While scalp recordings have contributed to understanding the psychological processes underlying the N400, they have been limited in identifying its neural basis. However, recent intracranial ERP recordings and fMRI studies have shed light on the crucial role of the anterior medial temporal lobe (AMTL) in semantic information processing. These findings suggest that the N400 partially represents activity in the AMTL structures. To investigate the neural underpinnings of the N400 effect, we simultaneously recorded ERPs and event-related fMRI during a semantic priming paradigm in a sample of 12 young, healthy subjects. Additionally, we collected ERPs and structural brain data from older healthy adults and patients with amnestic mild cognitive impairment (aMCI), a population characterized by neurodegenerative changes in the AMTL. In our fMRI results, we identified bilateral loci in the AMTL as the global maxima. Employing an EEG-informed fMRI analysis, we explored trial-to-trial fluctuations in semantic processing by linking single-trial N400 amplitudes to the Blood Oxygen Level Dependent (BOLD) signal. This approach provided the first direct evidence linking the N400 recorded at the scalp level to the corresponding BOLD signal in the AMTL. Consistent with these findings, patients with aMCI exhibited a diminished N400 effect compared to healthy older adults. Furthermore, voxel-based morphometry analysis revealed a correlation between the magnitude of the N400 effect and the integrity of the AMTL. By integrating data from simultaneous EEG-fMRI, and patient studies, our research advances our understanding of the neural substrate of the N400 and highlights the critical involvement of the AMTL in semantic processing.