Quality of life (QoL) is lower in adults on the autism spectrum (AS) compared with typically developing (TD) adults. In this context, recent studies have examined the role of depression and anxiety in reducing QoL in AS adults. The aim of this study was to (1) replicate these findings of lower QoL and (2) assess the negative influence of depressive and anxious symptoms on QoL in an adult AS (N = 86) and TD (N = 87) German sample with a broad age range (18–70 years). For this, we used questionnaires that have been validated for the AS and TD population: the World Health Organization Quality of Life Brief Version, the Autism-Specific QoL items, and the Hospital Anxiety and Depression Scale. We replicated previous findings and extended them to autism-specific QoL. Our AS sample had lower QoL compared with the TD adults. However, depressive symptoms were the largest contributor to lower QoL in both samples, more so than group membership and anxious symptoms. We conclude that interventions to improve QoL in AS adults should specifically target depressive symptoms and for this, improvements to the diagnostic process and treatment of depression in AS are necessary.