The deep understanding of the behaviors of traffic participants is essential to guarantee the safety of automated vehicles (AV) in mixed traffic with vulnerable road users (VRUs). Precise trajectory prediction of traffic participants can provide reasonable solution space for motion planning of AV. Early works mainly focused on handcrafting the feature representation and designing complicated architectures in deep learning-based prediction models. However, these approaches overlooked the fact that different road users perceive the safety of the same interaction differently and also exhibit heterogeneous risk-taking styles. In this paper, we will develop a model for trajectory prediction based on risk-taking styles. The model accounts for the expected positions and occupancy of traffic participants in the surrounding environment. It consists of two sequential steps: risk-taking styles of multi-modal road users under interactive scenes are first clustered, and then reformulated in the heterogeneous graph model for trajectory prediction. The model is validated by the driving data collected on the urban road using a public dataset. Comparative experiments demonstrate that the proposed method can predict the trajectory of traffic participants much more accurately than the state-of-the-art methods.