Humans' ability to gauge another person's body size from their voice alone may serve multiple functions ranging from threat assessment to speaker normalization. However, how this ability is acquired remains unknown. In two experiments we tested whether sighted, congenitally blind and late blind adults could accurately judge the relative heights of women from paired voice stimuli, and importantly, whether errors in size estimation varied with task difficulty across groups. Both blind (n = 56) and sighted (n = 61) listeners correctly judged women's relative heights on approximately 70% of low difficulty trials, corroborating previous findings for judging men's heights. However, accuracy dropped to chance levels for intermediate difficulty trials and to 25% for high difficulty trials, regardless of the listener's sightedness, duration of vision loss, sex, or age. Thus, blind adults estimated women's height with the same degree of accuracy, but also the same pattern of errors, as did sighted controls. Our findings provide further evidence that visual experience is not necessary for accurate body size estimation. Rather, both blind and sighted listeners appear to follow a general rule, mapping low auditory frequencies to largeness across a range of contexts. This sound-size mapping emerges without visual experience, and is likely very important for humans.