Repeated hospitalizations are a characteristic of severe disease courses in patients with affective disorders (PAD). To elucidate how a hospitalization during a nine-year follow-up in PAD affects brain structure, a longitudinal case-control study (mean [SD] follow-up period 8.98 [2.20] years) was conducted using structural neuroimaging. We investigated PAD (N = 38) and healthy controls (N = 37) at two sites (University of Münster, Germany, Trinity College Dublin, Ireland). PAD were divided into two groups based on the experience of in-patient psychiatric treatment during follow-up. Since the Dublin-patients were outpatients at baseline, the re-hospitalization analysis was limited to the Münster site (N = 52). Voxel-based morphometry was employed to examine hippocampus, insula, dorsolateral prefrontal cortex and whole-brain gray matter in two models: (1) group (patients/controls)×time (baseline/follow-up) interaction; (2) group (hospitalized patients/not-hospitalized patients/controls)×time interaction. Patients lost significantly more whole-brain gray matter volume of superior temporal gyrus and temporal pole compared to HC (p _FWE = 0.008). Patients hospitalized during follow-up lost significantly more insular volume than healthy controls (p _FWE = 0.025) and more volume in their hippocampus compared to not-hospitalized patients (p _FWE = 0.023), while patients without re-hospitalization did not differ from controls. These effects of hospitalization remained stable in a smaller sample excluding patients with bipolar disorder. PAD show gray matter volume decline in temporo-limbic regions over nine years. A hospitalization during follow-up comes with intensified gray matter volume decline in the insula and hippocampus. Since hospitalizations are a correlate of severity, this finding corroborates and extends the hypothesis that a severe course of disease has detrimental long-term effects on temporo-limbic brain structure in PAD.