This paper examines whether measurement non-invariance introduces bias into comparisons of means across language groups, leading to spurious findings. A probabilistic sample was taken from three large cities in a German federal state (N = 542), and mail invitations to a web survey were sent out in two languages (German and Arabic). Mental and physical health were evaluated with the Short-Form Health Survey (SF-12). Participants were randomly assigned to either a baseline SF-12 version or an adapted version developed through cognitive pretesting. Reliability and validity were evaluated using Structural Equation Models (SEM) to ensure nomological and criterion validity and to confirm that the SF-12 versions were parallel. Multiple Indicators Multiple Causes (MIMIC) models were implemented. Substantial non-invariance was identified in the mental health subscale of the baseline version, leading to an overestimation of ethnic group latent mean differences. Measurement non-invariance can lead to overestimation of ethnic health disparities. In addition to ensuring construct validity and reliability, uniform measurement invariance should be established or controlled for before interpreting mean differences among groups.