We report an experimental study of temperature-dependent spectral line shapes of phonon sideband emission stemming from dark excitons in monolayer and bilayer WSe2. Using photoluminescence spectroscopy in the range from 4 to 100 K, we observe a pronounced asymmetry in the phonon-assisted luminescence from momentum-indirect exciton reservoirs. We demonstrate that the corresponding spectral profiles are distinct from those of bright excitons with direct radiative decay pathways. The line-shape asymmetry reflects thermal distribution of exciton states with finite center-of-mass momenta, characteristic for phonon sideband emission. The extracted temperature of the exciton reservoirs is found to generally follow that of the crystal lattice, with deviations reflecting overheated populations. The latter are most pronounced in the bilayer case and at lowest temperatures. Our results add to the understanding of phonon-assisted recombination of momentum-dark excitons and, more generally, establish means to access the thermal distribution of finite-momentum excitons in atomically thin semiconductors with indirect band gaps.