Hourglass-shaped magnetic excitation spectra have been detected in a variety of doped transition-metal oxides with stripelike charge order. Compared to the predictions of spin-wave theory for perfect stripes, these spectra display a different intensity distribution and anomalous broadening. Here we show, based on a comprehensive modeling for La _5/3Sr _1/3CoO ₄, how quenched disorder in the charge sector causes frustration, and consequently cluster-glass behavior at low temperatures, in the spin sector. This spin-glass physics, which is insensitive to the detailed nature of the charge disorder, but sensitive to the relative strength of the magnetic interstripe coupling, ultimately determines the distribution of magnetic spectral weight: The excitation spectrum, calculated using spin waves in finite disordered systems, is found to match in detail the observed hour-glass spectrum.