Trust in government and institutions is a cornerstone of functioning representative democracies. While previous research documented negative associations between social media use as a news source and institutional trust, temporal dynamics and even basic moderating factors remain underexplored. We aim to contribute evidence on these issues by asking: How is the media use-trust association shaped by moderators like traditional media use, age and partisanship, and does it have a time component? For this, we analyze a unique comparative dataset comprising 20 representative panel waves in the United States and Germany, each consisting of at least 1500 participants (totaling in N = 16,679 in the US, N = 11,277 in Germany). We find that associations of trust and social media use are crucially moderated by traditional media use, particularly for younger people. In Germany, evidence from our specific survey structure suggests that lower trust temporally precedes higher social media use, a relationship that has not been documented before. At the same time, media use patterns differ between opposition and government party supporters, and these differences increase with age. Further, associations of social media use and government distrust are most pronounced among older participants and opposition supporters. A deeper understanding of these dynamics and heterogeneities is essential for democracies worldwide, particularly in the context of increasing social media use.