Wording Effects in Assessment: Missing the Trees for the Forest

Ponce, Fernando P.; Irribarra, David Torres; Vergés, Álvaro; Arias, Víctor

doi:10.1080/00273171.2021.1925075

Cited by 11 publications

(30 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The present study aimed to examine the apparent trait-like nature of the wording effect using the analytical strategy proposed by Ponce et al (2021). As noted above, the consistency between scales and the temporal stability of wording effects are the most relevant evidence to demonstrate the substantive nature of these effects.…”

Section: The Present Studymentioning

confidence: 96%

“…The bifactor model's tendency to overfit is due to its functional form and complexity (Bonifay & Cai, 2017), resulting in excellent fit indices interpreted as the underlying structure for balanced scales. The most significant risk is that the specific wording factors do not have an adequate interpretation, given that they can be estimated from improbable and implausible responses that introduce multidimensionality capitalized by the specific factors (Ponce et al, 2021).…”

Section: Wording Effects and Their Substantive Versus Ephemeral Naturementioning

confidence: 99%

“…After removing these participants from the analysis, they found an essentially unidimensional solution for both scales. Ponce et al (2021) noted that with both techniques, once the cases that contribute to heterogeneity are detected, there is no clear way to substantively characterize them or report the results between the groups comparably due to the absence of common parameters that allow equating the scales on which the results of the resulting noninvariant classes were expressed.…”

Section: Approaches Based On Population Heterogeneitymentioning

confidence: 99%

“…To illustrate their point, they identified and removed a group of participants who responded differentially to direct and reverse items of the RSES and found a substantive improvement in the single-factor model fit that was consistent with the impact of wording effects reported in the literature. More importantly, they also noted that this analytical approach enables a deeper analysis of the definitions and suggestions made in the literature for examining the language effect as a response style (for a more detailed description, see Ponce et al, 2021).…”

Section: Approaches Based On Population Heterogeneitymentioning

confidence: 99%

See 3 more Smart Citations

The ephemeral nature of wording effects

Ponce¹,

Irribarra²,

Vergés³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This article examines the analysis and interpretation of wording effects associated with using direct and reverse items in psychological assessment. Previous research using bifactor models has suggested a substantive nature of this effect. The present study uses mixture modeling to systematically examine an alternative hypothesis to surpass acknowledged limitations in the bifactor modeling approach. In Studies 1 and 2, we examined the presence of participants who exhibited wording effects and evaluated their impact on the dimensionality of Rosenberg’s Self-Esteem (RSES) and the Revised Life Orientation Test (LOT-R), confirming the ubiquity of wording effects in scales containing direct and reverse items. Then, using the data for both scales (n = 5953), we found that while there was a significant association between wording factors (Study 3a), there was also a low proportion of participants simultaneously exhibiting asymmetric responses in both scales (Study 3b). Similarly, despite finding both longitudinal invariance and temporal stability of this effect in three waves (n = 3712, Study 4a), the proportion of participants consistently identified with symmetric responses was low over time (Study 4b), reflected in lower transition parameters compared to the other patterns of profiles examined. These findings provide strong support for the notion of an ephemeral nature underlying wording effects. Discussion focuses on alternative hypotheses to understand these findings and emphasizes, contrary to popular misconceptions, the utility of including inverse items in psychological assessment.

show abstract

Section: The Present Studymentioning

confidence: 96%

Section: Wording Effects and Their Substantive Versus Ephemeral Naturementioning

confidence: 99%

Section: Approaches Based On Population Heterogeneitymentioning

confidence: 99%

Section: Approaches Based On Population Heterogeneitymentioning

confidence: 99%

See 2 more Smart Citations

The ephemeral nature of wording effects

Ponce¹,

Irribarra²,

Vergés³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This problem extends to the structure of personality survey data, usually investigated by factor analysis. Factor analysis assumes that all data come from the same population (Ponce et al, 2021). However, if there are C/IE respondents in the sample, this assumption is no longer true.…”

Section: The Present Studymentioning

confidence: 99%

How a Few Inconsistent Respondents Can Confound the Structure of Personality Survey Data

Arias

Ponce

Martínez-Molina

2023

European Journal of Psychological Assessment

Self Cite

View full text Add to dashboard Cite

Abstract. In survey data, inconsistent responses due to careless/insufficient effort (C/IE) can lead to problems of replicability and validity. However, data cleaning prior to the main analyses is not yet a standard practice. We investigated the effect of C/IE responses on the structure of personality survey data. For this purpose, we analyzed the structure of the Core-Self Evaluations scale (CSE-S), including the detection of aberrant responses in the study design. While the original theoretical model of the CSE-S assumes that the construct is unidimensional ( Judge et al., 2003 ), recent studies have argued for a multidimensional solution (positive CSE and negative CSE). We hypothesized that this multidimensionality is not substantive but a result of the tendency of C/IE data to generate spurious dimensions. We estimated the confirmatory models before and after removing highly inconsistent response vectors in two independent samples (6% and 4.7%). The analysis of the raw samples clearly favored retaining the two-dimensional model. In contrast, the analysis of the clean datasets suggested the retention of a single factor. A mere 6% C/IE response rate showed enough power to confound the results of the factor analysis. This result suggests that the factor structure of positive and negative CSE factors is spurious, resulting from uncontrolled wording variance produced by a limited proportion of highly inconsistent response vectors. We encourage researchers to include screening for inconsistent responses in their research designs.

show abstract