Mathematical ability is heritable and related to several genes expressing proteins in the brain. It is unknown, however, which intermediate neural phenotypes could explain how these genes relate to mathematical ability. Here, we examined genetic effects on cerebral cortical volume of 3–6-year-old children without mathematical training to predict mathematical ability in school at 7–9 years of age. To this end, we followed an exploration sample (n = 101) and an independent replication sample (n = 77). We found that ROBO1, a gene known to regulate prenatal growth of cerebral cortical layers, is associated with the volume of the right parietal cortex, a key region for quantity representation. Individual volume differences in this region predicted up to a fifth of the behavioral variance in mathematical ability. Our findings indicate that a fundamental genetic component of the quantity processing system is rooted in the early development of the parietal cortex.