How a Few Inconsistent Respondents Can Confound the Structure of Personality Survey Data

Abstract

Abstract. In survey data, inconsistent responses due to careless/insufficient effort (C/IE) can lead to problems of replicability and validity. However, data cleaning prior to the main analyses is not yet a standard practice. We investigated the effect of C/IE responses on the structure of personality survey data. For this purpose, we analyzed the structure of the Core-Self Evaluations scale (CSE-S), including the detection of aberrant responses in the study design. While the original theoretical model of the CSE-S assumes that the construct is unidimensional ( Judge et al., 2003 ), recent studies have argued for a multidimensional solution (positive CSE and negative CSE). We hypothesized that this multidimensionality is not substantive but a result of the tendency of C/IE data to generate spurious dimensions. We estimated the confirmatory models before and after removing highly inconsistent response vectors in two independent samples (6% and 4.7%). The analysis of the raw samples clearly favored retaining the two-dimensional model. In contrast, the analysis of the clean datasets suggested the retention of a single factor. A mere 6% C/IE response rate showed enough power to confound the results of the factor analysis. This result suggests that the factor structure of positive and negative CSE factors is spurious, resulting from uncontrolled wording variance produced by a limited proportion of highly inconsistent response vectors. We encourage researchers to include screening for inconsistent responses in their research designs.

Reliability and factorial validity of the Core Self-Evaluations Scale: A meta-analytic investigation of wording effects

“…Because the content facets of positive and negative CSE perfectly align with the wording of the items, it has been suggested that the CSES does not represent substantively different traits. Rather, its dimensionality is often distorted by artifacts stemming from the use of positively and negatively worded items (e.g., Arias et al, 2022;Gu et al, 2015;Henderson & Gardiner, 2019;Schmalbach et al, 2020). A frequently observed phenomenon in self-report instruments is systematic variance captured by negatively worded items that can present itself as an additional factor beyond the focal construct (e.g., DiStefano & Motl, 2006;Koutsogiorgi et al, 2021).…”

“…Although developed to represent a single latent trait, factor analytic investigations of the CSES often favored multi-as compared to unidimensional measurement models (e.g., Arias et al, 2022;Henderson & Gardiner, 2019;Mäkikangas et al, 2018;Schmalbach et al, 2020;Sun et al, 2017;Zenger et al, 2015). These results have been interpreted either from a substantive point of view as reflecting different content facets of CSE (Mäkikangas et al, 2018;Zenger et al, 2015) or as the result of methodological artifacts stemming from the item wording (Arias et al, 2022;Schmalbach et al, 2020). The present study contributes to this debate by presenting meta-analytic evidence on the psychometric properties of the CSES.…”

“…Extensive research on the CSES attested to its good reliability with an average coefficient alpha of .84 (Ock et al, 2021) and usefulness for predicting various outcomes such as psychological and physical health (Turska & Stępień-Lampa, 2021) or income and number of promotions (Stumpp et al, 2010). Despite the substantial evidence for the validity of the sum score, the internal structure of the CSES is still disputed (e.g., Arias et al, 2022;Gu et al, 2015;Henderson & Gardiner, 2019;Mäkikangas et al, 2018;Schmalbach et al, 2020;Sun et al, 2017;Zenger et al, 2015). In line with its original conception, Judge and colleagues (2003) seem to provide evidence that a unidimensional confirmatory factor model had the best fit for the CSES as compared to more complex measurement models.…”

“…Authors of eligible studies reporting factor loading patterns of the CSES were contacted by email and asked to share the correlation matrix or raw data of their study. Three authors were responsive (Arias et al, 2022;Gu et al, 2015;Zenger et al, 2015). This literature search and screening process (see Figure S1 in the online supplement for an overview) resulted in 37 sources with 41 samples that could be included in the meta-analytic database (see Table 1).…”

The ephemeral nature of wording effects.

“…However, several studies have shown differences in the scores of direct items and recoded reverse items (also known as wording effects). This effect has been described as a tendency to respond differentially to both subsets, producing ubiquitous alterations in data quality that can induce (a) spurious relationships between variables that would not otherwise be correlated (Arias et al, 2022; Huang et al, 2015), (b) an increase/decrease in internal consistency (Roszkowski & Soven, 2010; Wood et al, 2017), (c) mismatch in unidimensional models or the emergence of method factors associated with direct or reverse items (Greenberger et al, 2003; Woods, 2006), (d) alterations in discriminative capacity (Rodebaugh et al, 2004), or (e) a systematic bias that undermines the effectiveness of experimental manipulations (Maniaci & Rogge, 2014), to list a few examples. In this respect, the interpretation and statistical control of these systematic elements present conceptual and analytical challenges for the quality of the evaluation when using self-report measures.…”