Introduction: Accurate health care evaluation using quality indicators (QIs) is of vital clinical importance for a quality-oriented health care system. The QUALIFY tool is the current research standard for assessing QIs of health care; however, its biometric properties in psychiatry have not yet been evaluated empirically. Our aim was to evaluate the internal consistency and structure of QUALIFY. Methods: This study applies a literature-based post-hoc analytical design to a sample of 289 QIs of mental health care. First, the indicators were assessed on the basis of nineteen ordinal QUALIFY criteria as a single measuring tool. Second, using Cronbach's alpha the internal consistency of the measuring tool was evaluated and the structure of QUALIFY using an explorative principal component analysis was tested. Results and Discussion: The tool showed an acceptable internal consistency (Cronbach's α = 0.75), with three criteria (consideration of potential risks/side effects when using the indicator, implementation barriers taken into account, and the ability to influence the indicator) being inconsistent with the full scale. If these three criteria were not taken into account, the tool had a good internal consistency (Cronbach's α = 0.81). The QUALIFY structural matrix comprises three components, one of which reflected six from eight original quality criteria of the scientific category. The other two components represent the semiotic structure of the QIs. Conclusion: QUALIFY is an internally inconsistent instrument, which may be useful to assess mental health care QIs. The information about the structure of QUALIFY can be applied for the purposes of research planning as well as the interpretation and development of QIs.