Autism spectrum disorder (ASD) might be conceptualized as an essentially dimensional, categorical, or hybrid model. Yet, current empirical studies are inconclusive and the latent structure of ASD has explicitly been examined only in a few studies. The aim of our study was to identify and discuss the latent model structure of behavioral symptoms related to ASD and to address the question of whether categories and/or dimensions best represent ASD symptoms. We included data of 2920 participants (1–72 years of age), evaluated with the Autism Diagnostic Observation Schedule (Modules 1–4). We applied latent class analysis, confirmatory factor analysis, and factor mixture modeling and evaluated the model fit by a combination of criteria. Based on the model selection criteria, the model fits, the interpretability as well as the clinical utility we conclude that the hybrid model serves best for conceptualization and assessment of ASD symptoms. It is both grounded in empirical evidence and in clinical usefulness, is in line with the current classification system (DSM-5) and has the potential of being more specific than the dimensional approach (decreasing false positive diagnoses).