Why do judgments on different person-descriptive attributes correlate with one another? A conceptual analysis with relevance for most psychometric research

Abstract

Most psychometric research relies heavily on patterns of correlations between items on which perceivers describe targets. Apart from the targets’ actual (“substantive”) characteristics, this type of data has been shown to also reflect a number of other, non-substantive sources of variation. This is problematic because each of these (semantic redundancy, attitudes, formal response styles) may all by itself account for correlations among items, which may then be misinterpreted in terms of substantive effects. We present an integrative theoretical account of how these non-substantive influences may affect the pattern of relationships between items. We also point out how this is relevant to the validity of conclusions drawn in research on “general factors” (of personality, psychopathology, and personality pathology) and on “network models”. Furthermore, we discuss various ways of dealing with the problem, which is necessary before any correlations between items may be interpreted in terms of substantive associations between the targets’ actual characteristics.

Unique variable analysis: A network psychometrics method to detect local dependence

“…The other two factors split the theoretical negative schizotypy factor into affective (8 items) and social anhedonia (18 items) factors. This unexpected separation of the negative schizotypy factor was perhaps due to minor factors of semantically similar content rather than substantive causal differences (Leising et al, 2020). Psychometric network models are not the only models affected by redundancy though.…”

“…Some of the more consequential effects may appear when attempting to measure broad attributes such as personality traits where facets contain some redundancy but not completely-that is, some items that make up a facet are redundant while others are not (Christensen, Golino, & Silvia, 2020). Oftentimes, these issues appear because of shared semantic similarity where some items are slight variations in wording and content (Leising et al, 2020;Rosenbusch, Wanders, & Pit, 2020). Undoubtedly, semantic similarity between items will create more internally consistent scales but at the cost of introducing confounds into the validity of the measurement.…”

“…One variable on each factor, which we refer to as target variables, also loaded onto a third factor. In practical terms, such a factor might represent shared semantic similarities between two variables (e.g., "I like to go to parties" and "I like to go to art museums"; Leising et al, 2020;Rosenbusch, Wanders, & Pit, 2020). They incrementally manipulated the loadings of the target variables between the shared and non-shared factors, forming a gradient of shared to non-shared factor effects on the centrality measures.…”

“…Incidentally, Cronbach's α is sensitive to redundancy and will tend to overestimate internal consistency in these conditions (Green & Yang, 2009). We define redundancy broadly meaning the potential for a shared substantive cause (e.g., single attribute), shared semantic reference (e.g., similar item phrasing, similar item content; Rosenbusch, Wanders, & Pit, 2020), and general response tendency (e.g., social desirability; see Leising et al, 2020 for other potential causes). In classical and modern test theory, redundancy is generally thought to be beneficial for test construction because it increases test homogeneity and minimizes idiosyncratic causes (i.e., reduces measurement error; Lord & Novick, 1968).…”

A less evaluative measure of Big Five personality: Comparison of structure and criterion validity

“…Researchers and practitioners have long endeavored to understand and counter the detrimental effects of socially desirable responding bias on the structure (Saucier, 2002;Schmit & Ryan, 1993;Zickar & Robie, 1999) and criterion validity of personality assessments (Douglas, McDaniel, & Snell, 1996;Jeong, Christiansen, Robie, Kung, & Kinney, 2017;Morgeson et al, 2007;Rothstein & Goffin, 2006;Tett, Jackson, & Rothstein, 1991;Topping & O'Gorman, 1997). Similarly, debates about whether agreement with socially desirable items reflects substance or bias can be seen in multiple personality literatures including discussion of impression management scales (de Vries, Zettler, & Hilbig, 2014;Nederhof, 1985;Paulhus, 1984;Uziel, 2010), personality modelling (Anglim, Lievens, Everton, Grant, & Marty, 2018;Davies, Connelly, Ones, & Birkland, 2015;Leising, Burger, et al, 2020), high-stakes assessment (Douglas et al, 1996;Hough, 1997;Jeong et al, 2017;Mueller-Hanson, Heggestad, & Thornton III, 2003), and the general factor of personality (Anglim, Morse, Dunlop, Minbashian, & Marty, 2020;Anusic, Schimmack, Pinkus, & Lockwood, 2009;de Vries et al, 2014;Musek, 2007;Revelle & Wilt, 2013;Van der Linden, te Nijenhuis, & Bakker, 2010). One promising approach for contributing to these debates, and potentially reducing the effect of socially desirable responding bias, is item neutralization (Bäckström, Björklund, & Larsson, 2009.…”