Biodiversity loss can destabilize ecosystem functioning. How biodiversity–stability relationships are interlinked across trophic levels remains poorly investigated, however, limiting our ability to predict ecosystem-level consequences of declining biodiversity. Here, we analyze the drivers of multi-year herbivore community stability—as a key connector between primary producers and higher trophic levels—and its coupling with host tree diversity and growth stability along a subtropical tree diversity gradient. Phylogenetic diversity, abundance asynchrony and population stability of herbivores emerge as key intra-community regulators of herbivore temporal stability. These regulators, in turn, are strongly affected by changes in tree species richness through tree functional diversity, tree growth asynchrony, and tree growth population stability. Importantly, accounting for herbivore dietary specialization unveils clear stabilizing effects of tree species richness on the community stability of specialists but not of generalists. For the overall herbivore community, higher tree richness results in less stable abundance dynamics. Our findings suggest that biodiversity loss will propagate bottom-up to affect the stability of communities at higher trophic levels, and particularly destabilize communities of more vulnerable specialists. Global change and plantation management may thus also compromise biodiversity conservation by reducing abundance and species richness stability of higher trophic levels.