In randomized experiments, inventories with reverse-keyed items are compared with inventories in which all the items are either positively or negatively associated with the underlying concept. The results show that with reverse keying, a control of the potential bias was not sufficient; likewise, the factorial structure, reliability, and validity were negatively affected. An eye-tracking study revealed that respondents did not process information more deeply with the reverse-keyed items than with the other forms. Respondents seemed to find it difficult to process reverse keying mentally so that it is not sufficient to use it in heterogeneous samples and with short inventories.