Adolescence is a period of major brain reorganization shaped by biologically timed and by environmental factors. We sought to discover linked patterns of covariation between brain structural development and a wide array of these factors by leveraging data from the IMAGEN study, a longitudinal population-based cohort of adolescents. Brain structural measures and a comprehensive array of non-imaging features (relating to demographic, anthropometric, and psychosocial characteristics) were available on 1476 IMAGEN participants aged 14 years and from a subsample reassessed at age 19 years (n = 714). We applied sparse canonical correlation analyses (sCCA) to the cross-sectional and longitudinal data to extract modes with maximum covariation between neuroimaging and non-imaging measures. Separate sCCAs for cortical thickness, cortical surface area and subcortical volumes confirmed that each imaging phenotype was correlated with non-imaging features (sCCA r range: 0.30–0.65, all P_FDR < 0.001). Total intracranial volume and global measures of cortical thickness and surface area had the highest canonical cross-loadings (|ρ| = 0.31−0.61). Age, physical growth and sex had the highest association with adolescent brain structure (|ρ| = 0.24−0.62); at baseline, further significant positive associations were noted for cognitive measures while negative associations were observed at both time points for prenatal parental smoking, life events, and negative affect and substance use in youth (|ρ| = 0.10−0.23). Sex, physical growth and age are the dominant influences on adolescent brain development. We highlight the persistent negative influences of prenatal parental smoking and youth substance use as they are modifiable and of relevance for public health initiatives.