Background: The COVID-19 pandemic increased psychological distress among health care workers (HCWs). Those with informal caregiving responsibilities (ICs) may be especially vulnerable, but data remain limited. Methods: In a multicenter online survey conducted across four time points (T1–T4: 2020–2022) within the German healthcare system, HCWs with ICs were compared to those without (comparison group, CG). Psychological distress was assessed using validated measures of anxiety (Generalized Anxiety Disorder/GAD-2) and depression (Patient Health Questionnaire/PHQ-2). Group differences were analyzed using non-parametric Mann–Whitney U and chi-square tests. Longitudinal logistic regression analyses examined the impact of IC on psychological distress, controlling for sociodemographic, occupational, and psychological factors. A moderation analysis tested whether fear of infecting relatives influenced the salutogenic effect of optimism. All analyses were performed with multiply imputed data and a retrospective sample size justification was conducted. Results: ICs were more likely to be female, older, work in occupations other than medicine, work part-time, have children, and have more than 6 years of work experience. While ICs showed significantly higher PHQ-4 anxiety and depression than CG at T1 and T2, no significant differences could be shown for the other time points. IC emerged as a significant risk factor for increased psychological distress longitudinally, even after controlling for confounders. The protective effect of optimism was moderated by fear of infecting relatives at T1. Conclusion: HCWs with informal caregiving duties represent a vulnerable subgroup with increased psychological distress, especially during the early pandemic. This group may benefit from targeted support (e.g., flexible schedules, protective equipment, psychological interventions). Limitations include lack of pre-pandemic data and reliance on self-report. Findings underscore the importance of acknowledging and addressing overlapping care burdens in future health crises.