Previous studies have shown that people are good at rapidly estimating ensemble summary statistics, such as the mean size of multiple objects. In the present study, we tested whether these average estimates are based on "raw" retinal representations (proximal sizes) or on how items should appear based on context, such as the viewing distance (distal sizes). In our experiments, observers adjusted the mean size of multiple objects presented at various apparent distances through a stereoscope. In Experiment 1, all items were shifted in depth by the same amount while the adjustable probe stayed at the fixed middle position. We found that presenting ensembles in an apparently remote plane made observers overestimate the mean size, which is consistent with angular sizes being rescaled to distance. In Experiment 2, we presented individual sizes in different planes. While angular sizes and apparent distances were kept controlled across conditions, we only manipulated correlations between them. These manipulations affected the precision of size averaging in line with changes in the range of apparent rather than angular sizes. This pattern is possible only if the visual system rescales each individual size to its distance prior to averaging. Our finding demonstrates that ensemble summaries of basic features, such as size, can be based on quite elaborated representations of multiple objects. We also discuss important implications for size constancy.