Regular recreational use of 3,4-methylenedioxymethamphetamine (MDMA, "Ecstasy") has been consistently linked to verbal memory dysfunctions, whose neurobiological origins remain unclear. Although neurochemical, structural, and functional alterations resulting from repeated MDMA intake have been identified in user populations, only limited knowledge exists regarding potential interrelationships among these components and their implications for mnemonic functions. The present study, thus, first examined the association of MDMA use with grey matter volume and cortical thickness. Subsequently, structural brain alterations were related to verbal memory performance and atlas-derived cerebral serotonin (5-HT) receptor expression patterns. Our data, involving 122 participants (61 MDMA users, 61 age-, sex-, and education-matched MDMA-naive controls), revealed a robust reduction of grey matter volume within hippocampal regions and impaired verbal learning, short-term recall after interference, long-term recall, and recognition performance in MDMA users compared to controls. Self-reported MDMA use in the past six months correlated with several memory performance scores. A moderate inverse correlation across all participants was observed between hippocampal volumes and verbal long-term memory. Correspondingly, hippocampal CA1 grey matter volume was related to MDMA use intensity in the last months as indicated by MDMA hair concentrations. The extent of cortical and subcortical grey matter differences between groups correlated with atlas-based 5-HT1A, 5-HT2A, and 5-HT4 receptor densities, suggesting a serotonergic basis for mnemonic and structural alterations in MDMA users. Together, these findings highlight a multidimensional origin of memory dysfunction in MDMA users, where alterations in the chemoarchitecture of the brain may affect behavior, possibly via influences on brain grey matter structures.