In recent years, Central European forests and trees have been severely suffered from drought events in the growing season. To examine the tree growth response to drought stress, measurements of annual shoot lengths (ASL) of various tree species with distinct ages and habitats were taken in southeast Germany. The index of relative ASL increase (RAI), which is a standardized ASL increase, was calculated to indicate the growth rate of various tree species between the non-drought (2015–2017) and the drought period (2018–2020). The results showed the ASLs of most early-young trees (< 10 years) significantly showed a decreasing tendency in the drought period, whereas the middle-young (11–15 years) and late-young trees (> 16 years) presented a relatively stable trend. The ring-porous species with anisohydric behavior were likely to have higher RAIs associated with higher drought tolerance than diffuse-porous species with isohydric behavior. However, tree growth rate did not correlate with drought tolerance (resistance) of tree species indicated by leaf turgor loss point (Ψ_tlp, MPa). Therefore, this study suggested parameters, such as growth rate, age phases, and microhabitats, should be considered to predict tree species’ drought tolerance other than Ψ_tlp. Finally, we emphasized tree shoot growth responses to drought were affected by multiple internal and external factors, and strongly depended on species- and site-specific characteristics, such as age phases, xylem structure, hydraulic strategy, microhabitat, grafting effect, and competition. ASL measurement can be recommended to be an easy, fast, and effective method to detect tree growth rate and response to environmental stress.