As a vital stormwater pollution source, the pollutants associated with road-deposited sediment (RDS) have become a growing concern in urban water management. Green infrastructure has exhibited great potential in stormwater pollution mitigation, but is not comprehensively understood yet due to the influences of complex RDS-associated pollutant migration processes (i.e., build-up, wash-off, and discharge). In this study, a city-scale hydraulic and water quality model was used to analyze the migration and removal processes of four RDS-associated pollutants (total suspended solids (TSS), chemical oxygen demand (COD), total nitrogen (TN), and total phosphorus (TP)) under different hydrological patterns, land-cover types, and green infrastructure installation locations. Results show that the antecedent dry-weather period was the main factor influencing RDS build-up, while the precipitation pattern strongly impacted RDS wash-off, discharge, and removal. The downstream-installed green infrastructures reduced the RDS-induced stormwater pollution by up to 68% and relieved the pollution-mitigation pressure of the studied drainage networks by almost 50%. The TSS and COD removal rates were higher (62.22–68.09%) near green space, while those of TN and TP were higher around buildings and roads (40.00–62.50%). Sensitivity analysis indicated that seven parameters regarding the surface layer characteristics and soil texture class strongly impacted the pollution-mitigation performance among the 31 technical parameters of green infrastructure. The results of this study would assist urban water management by optimizing green infrastructure for stormwater pollution mitigation.