Indigenous agroforestry systems in tropical mountainous environments provide crucial ecosystem services, but these ecosystems are also facing some challenges. A loss of diversity and native tree species in the overstory layer has been a growing concern in agroforestry worldwide, yet the drivers behind it remain inadequately understood. We hypothesize that the choice of overstory tree species is closely linked to the ecosystem services required by farmers, their livelihood strategy, and the salient features of each system. We, therefore, investigated four different farming systems in the mountains of northeastern Tanzania, i.e., the Kihamba on Mt. Kilimanjaro, Ginger agroforestry in the South Pare mountains, and Miraba and Mixed spices agroforestry in the West and East Usambara. In 82 farms, we collected data on the structure, tree species composition (both native and non-native), diversity, and associated provisioning ecosystem services as identified by smallholder farmers. Our results indicate that although all studied systems are multi-layered with three or four vertical layers, they have notable differences in their salient features concerning structure, composition, and diversity. The unique climate, landscape setting, soil, historical background, and economic opportunities that exist in each region contribute to those differences. Our findings indicate that the Kihamba system had the highest number of native tree species, and the largest diversity in species used for provisioning services, followed by Ginger agroforestry. No native species were used in Miraba or Mixed spices agroforestry, where a limited number of non-native tree species are planted mainly for fuel and timber or as a crop, respectively. Our findings regarding reported provisioning ES corroborate our hypothesis and imply that policies to increase resilience and restore the native tree species cover of the agroforestry systems of Tanzania can only be successful if knowledge of the ES potential of native species is increased, and interventions are tailored to each system’s ES needs for conservation as well as livelihood.