Using Constrained Factor Mixture Analysis to Validate Mixed-Worded Psychological Scales: The Case of the Rosenberg Self-Esteem Scale in the Dominican Republic

Abstract

A common method to collect information in the behavioral and health sciences is the self-report. However, the validity of self-reports is frequently threatened by response biases, particularly those associated with inconsistent responses to positively and negatively worded items of the same dimension, known as wording effects. Modeling strategies based on confirmatory factor analysis have traditionally been used to account for this response bias, but they have recently become under scrutiny due to their incorrect assumption of population homogeneity, inability to recover uncontaminated person scores or preserve structural validities, and their inherent ambiguity. Recently, two constrained factor mixture analysis (FMA) models have been proposed by Arias et al. (2020) and Steinmann et al. (2021) that can be used to identify and screen inconsistent response profiles. While these methods have shown promise, tests of their performance have been limited and they have not been directly compared. Thus the objective of the current study was to assess and compare their performance with data from the Dominican Republic of the Rosenberg Self-Esteem Scale (N = 632). Additionally, as this scale had not yet been studied for this population, another objective was to show how using constrained FMAs could help in the validation of mixed-worded scales. The results indicated that removing the inconsistent respondents identified by both FMAs (≈8%) reduced the amount of wording effects in the database. However, whereas the Steinmann et al. method only cleaned the data partially, the Arias et al. (2020) method was able to remove the great majority of the wording effects variance. Based on the screened data with the Arias et al. method, we evaluated the psychometric properties of the RSES for the Dominican population, and the results indicated that the scores had good validity and reliability properties. Given these findings, we recommend that researchers incorporate constrained FMAs into their toolbox and consider using them to screen out inconsistent respondents to mixed-worded scales.

Dimensionality Assessment in the Presence of Wording Effects: A Network Psychometric and Factorial Approach

“…Thus, with data contaminated by wording effects the dimensionality estimates no longer reflect the underlying substantive dimensionality, but rather, they suggest a combination of the number of substantive and method factors. This finding is consistent with the wording effects empirical factor-analytic literature (García-Batista et al, 2021;Yang et al, 2018;Zhang & Savalei, 2016). However, novel findings of this study show that the impact of the wording effects on the dimensionality estimates is complex, and notably different for the PA and EGA retention methods.…”

“…As the magnitude of the wording effects increases, factor retention methods are likely to detect this systematic variance and suggest a latent dimensionality greater than the number of substantive factors (García-Batista et al, 2021;Yang et al, 2018;Zhang & Savalei, 2016).…”

Dimensionality assessment in the presence of wording effects: A network psychometric and factorial approach

“…Specifying less -underfactoringor more -overfactoring-dimensions than those present in the population can have detrimental effects on the quality of the factor solutions, including substantial error in the factor loading and factor score estimates, factor splitting, inadmissible solutions, and the emergence of uninterpretable factors (Auerswald & Moshagen, 2019). The inherent difficulties of dimensionality assessment are further exacerbated by the response biases related to wording effects, which tend to increase the latent dimensionality of observed scores (García-Batista et al, 2021;Kam, 2018;Schmalbach et al, 2020;Yang et al, 2018;Zhang & Savalei, 2016). This, in turn, can lead to long-standing controversies regarding the substantive or artifactual nature of dimensions underlying mixed-worded scales (Gnambs et al, 2018).…”

Detecting non-content-based response styles in survey data: An application of mixture factor analysis

“…The detrimental effects of nCB data have been well documented: increased risk of type I error in decision-making between competing models, replication problems between studies with different proportions of nCB responses, spurious relationships between truly unrelated variables, artificial deflation or inflation of the internal consistency of data, appearance of factors other than those theoretically expected, obscured effects of experimental manipulation, and severe perturbations in the factorial structure of data (Arias et al, 2020a ; Curran, 2012 ; García-Batista et al, 2021 ; Goldammer et al, 2020 ; Huang et al, 2015 ; Johnson, 2005 ; Kam & Meyer, 2015 ; Maniaci & Rogge, 2014 ; Steinmann et al, 2022 ; Wood et al, 2017 ; Woods, 2006 ).…”

“…Various approaches have been proposed to model this inconsistency, usually by specifying additional factors (DiStefano & Motl, 2006 ; Eid, 2000 ; Gnambs et al, 2018 ; Horan et al, 2003 ; Marsh et al, 2010 ; Michaelides et al, 2016 ; Savalei & Falk, 2014 ; Tomás & Oliver, 1999 ; Weijters et al, 2013 ). However, recent studies using mixture models suggest that the phenomenon represented by the wording/method factor is not generalizable to the whole sample: on the contrary, a large proportion of spurious variance is due to a limited proportion of response vectors (Arias et al, 2020a ; García-Batista et al, 2021 ; Ponce et al, 2021 ; Reise et al, 2016 ; Steinmann et al, 2021 , 2022 ; Yang et al, 2018 ). Therefore, although modeling the wording variance helps to reveal the true structure of data, the estimates of the trait in the contaminated vectors remain biased, which may affect important properties of the data, such as the accuracy of the estimators, validity coefficients, or measurement invariance (Arias et al, 2020a ; Nieto et al, 2021 ; Tomás et al, 2015 ).…”