The ephemeral nature of wording effects

Abstract

This article examines the analysis and interpretation of wording effects associated with using direct and reverse items in psychological assessment. Previous research using bifactor models has suggested a substantive nature of this effect. The present study uses mixture modeling to systematically examine an alternative hypothesis to surpass acknowledged limitations in the bifactor modeling approach. In Studies 1 and 2, we examined the presence of participants who exhibited wording effects and evaluated their impact on the dimensionality of Rosenberg’s Self-Esteem (RSES) and the Revised Life Orientation Test (LOT-R), confirming the ubiquity of wording effects in scales containing direct and reverse items. Then, using the data for both scales (n = 5953), we found that while there was a significant association between wording factors (Study 3a), there was also a low proportion of participants simultaneously exhibiting asymmetric responses in both scales (Study 3b). Similarly, despite finding both longitudinal invariance and temporal stability of this effect in three waves (n = 3712, Study 4a), the proportion of participants consistently identified with symmetric responses was low over time (Study 4b), reflected in lower transition parameters compared to the other patterns of profiles examined. These findings provide strong support for the notion of an ephemeral nature underlying wording effects. Discussion focuses on alternative hypotheses to understand these findings and emphasizes, contrary to popular misconceptions, the utility of including inverse items in psychological assessment.