Environmental adversities constitute potent risk factors for psychiatric disorders. Evidence suggests the brain adapts to adversity, possibly in an adversity-type and region-specific manner. However, the long-term effects of adversity on brain structure and the association of individual neurobiological heterogeneity with behavior have yet to be elucidated. Here we estimated normative models of structural brain development based on a lifespan adversity profile in a longitudinal at-risk cohort aged 25 years (n = 169). This revealed widespread morphometric changes in the brain, with partially adversity-specific features. This pattern was replicated at the age of 33 years (n = 114) and in an independent sample at 22 years (n = 115). At the individual level, greater volume contractions relative to the model were predictive of future anxiety. We show a stable neurobiological signature of adversity that persists into adulthood and emphasize the importance of considering individual-level rather than group-level predictions to explain emerging psychopathology.