Oscillations of conductance observed in strong magnetic fields are a striking manifestation of the quantum dynamics of charge carriers in solids. The large charge carrier density in typical metals sets the scale of oscillations in both electrical and thermal conductivity, which characterize the Fermi surface. In semimetals, thermal transport at low-charge carrier density is expected to be phonon dominated, yet several experiments observe giant quantum oscillations in thermal transport. This raises the question of whether there is an overarching mechanism leading to sizable oscillations that survives in phonon-dominated semimetals. In this work, we show that such a mechanism exists. It relies on the peculiar phase-space allowed for phonon scattering by electrons when only a few Landau levels are filled. Our measurements on the Dirac semimetal ZrTe5 support this counterintuitive mechanism through observation of pronounced thermal quantum oscillations, since they occur in similar magnitude and phase in directions parallel and transverse to the magnetic field. Our phase-space argument applies to all low-density semimetals, topological or not, including graphene and bismuth. Our work illustrates that phonon absorption can be leveraged to reveal degrees of freedom through their imprint on longitudinal thermal transport.