The length of a vocal tract is reflected in the sound it is producing. The length of the vocal tract is correlated with body size and humans are very good at making size judgments based on the acoustic effect of vocal tract length only. Here we investigate the underlying mechanism for processing this main auditory cue to size information in the human brain. Sensory encoding of the acoustic effect of vocal tract length (VTL) depends on a time-stabilized spectral scaling mechanism that is independent of glottal pulse rate (GPR, or voice pitch); we provide evidence that a potential neural correlate for such a mechanism exists in the medial geniculate body (MGB). The perception of the acoustic effect of speaker size is influenced by GPR suggesting an interaction between VTL and GPR processing; such an interaction occurs only at the level of non-primary auditory cortex in planum temporale and anterior superior temporal gyrus. Our findings support a two-stage model for the processing of size information in speech based on an initial stage of sensory analysis as early as MGB, and a neural correlate of the perception of source size in non-primary auditory cortex.