Dimensionality Assessment in the Presence of Wording Effects: A Network Psychometric and Factorial Approach

Abstract

This study proposes a procedure for substantive dimensionality estimation in the presence of wording effects, the inconsistent response to regular and reversed self-report items. Although extensive empirical research had shown that wording effects negatively impact latent dimensionality estimates, there was scarce systematic research assessing the problem, and no validated solutions had been offered. The procedure developed consisted in subtracting the wording effects variance from the sample correlation matrix and then estimating the substantive dimensionality on the residual correlation matrix. This was achieved by estimating a random intercept factor with unit loadings for all the regular and unrecoded reversed items. The accuracy of the procedure was evaluated through a broad simulation study that manipulated six relevant variables and employed the exploratory graph analysis (EGA) and parallel analysis (PA) retention methods. The results indicated that combining the proposed procedure with EGA or PA achieved high accuracy in estimating the substantive latent dimensionality, but that EGA was superior. Additionally, the present findings shed light on the complex ways that wording effects impact the dimensionality estimates when the response bias in the data is ignored. A tutorial on substantive dimensionality estimation with the R package EGAnet is offered and practical guidelines for applied researchers.

The Basic Psychological Need Satisfaction and Frustration Scales probably do not validly measure need frustration.

“…As explained earlier, this complete confounding prevents us and other researchers from being able to dispositively determine how much item covariance is due to item-keying method bias versus due to the substantive distinction between satisfaction and frustration. In future research, one way to potentially ameliorate this inherent limitation is to craft reverse-worded twins for each of the 24 BPNSFS items and then use that doubled item pool in a series of studies evaluating the validity of the BPNSFS (for exemplar of this kind of approach, see Naragon-Gainey & DeMarree, 2017; also, for a newly developed exploratory graph analysis approach to be used with twinned items, see Garcia-Pardina et al, 2022).…”

Comparison of wellbeing structures based on survey responses and social media language: A network analysis

“…It is also known that survey‐based data may suffer from certain biases such as social desirability or recollection bias (Edwards, 1957; Shiffman et al, 1997) or wording effects (Schuman & Presser, 1996) as also found in our study. We have applied specific methods to solve wording‐related problems in partial correlation networks (residualEGA; Garcia‐Pardina et al, 2022), which appeared to remedy these problems effectively. In addition, although we have not reported similar type of problems in our social media language networks, existing evidence warns about potential biases to be found in social media data as well.…”

“…These wording effects and difficulties in replicating the originally hypothesised factor structure of Ryff's scales have previously been reported (Abbott et al, 2006; Burns & Machin, 2009; Sirigatti et al, 2009; Springer et al, 2006; Springer & Hauser, 2006; Triado et al, 2007). To attenuate for these wording effects in the survey‐based exploration and final networks, we used the ‘residualEGA’ from the EGAnet package (Garcia‐Pardina et al, 2022; for the details of the model, see Maydeu‐Olivares & Coffman, 2006). Using this method, we fitted a latent wording/method factor to account for wording effects, after which the EGA dimensions were modelled based on the remaining ‘residual correlation matrix’.…”