In double barreled questions (DBQs) respondents provide one answer to two questions. Assumptions how respondents treat DBQs and how DBQs impact measurement quality are tested in two randomized experiments. DBQs are compared with revisions in which one stimulus was retained while the other stimulus was skipped. The observed means and parameters when modeling latent variables differed among the versions. Metric and scalar measurement invariance was not given among the versions, and at least one single stimulus version was found to be associated with a higher validity. Response latencies did not differ among versions or respondents needed less time to respond to DBQs. The author concludes that respondents may understand the stimuli in a DBQ differently, and access one of them while disregarding the other, which can have an adverse effect on validity.