Urban forests and trees are affected by potential biotic and abiotic climate change impacts. To enhance urban forest adaptability and resilience to climate change impacts, tree species with high local climate adaptability and robust stress tolerance should be identified and selected. Climate events in the Shanghai area, such as late frost, chilling, heat waves, drought, typhoons, waterlogging, soil salinization, pests, and disease, directly or indirectly impact urban forests and trees. For urban tree species selection in the context of climate change, an assessment framework was proposed and applied to assess the climate change adaptability of 65 urban tree species in Shanghai using a method combined with quantitative data and qualitative descriptions. In this study, the climate types of tree species were divided into four groups according to annual mean temperature (AMT) and annual precipitation (AP): temperate, cool subtropical, warm subtropical, and moist subtropical species. The results showed that hardness, heat tolerance, chilling requirement, and drought tolerance were categorized as climate-related tolerances, while other tolerances were categorized as non-climate-related tolerances. The tree species’ optimal AMT and AP were significantly correlated with climate-related tolerances, but they did not respond to the non-climate-related tolerances. The warm subtropical species had higher stress tolerances than other climate types in the Shanghai area; therefore, the warm subtropical species with high tolerances were the most suitable alternatives for urban tree species selection with regards to Shanghai's climate change impacts. This study also found that the AMT optimum is a better index to reflect tree species’ climate-related tolerances rather than the AP optimum. Finally, the adaptability assessment framework of climate change impact will offer guidance for future-oriented urban forest management and urban tree species selection in Shanghai.