Thinking twice about sum scores

Abstract

A common way to form scores from multiple-item scales is to sum responses of all items. Though sum scoring is often contrasted with factor analysis as a competing method, we review how factor analysis and sum scoring both fall under the larger umbrella of latent variable models, with sum scoring being a constrained version of a factor analysis. Despite similarities, reporting of psychometric properties for sum scored or factor analyzed scales are quite different. Further, if researchers use factor analysis to validate a scale but subsequently sum score the scale, this employs a model that differs from validation model. By framing sum scoring within a latent variable framework, our goal is to raise awareness that (a) sum scoring requires rather strict constraints, (b) imposing these constraints requires the same type of justification as any other latent variable model, and (c) sum scoring corresponds to a statistical model and is not a modelfree arithmetic calculation. We discuss how unjustified sum scoring can have adverse effects on validity, reliability, and qualitative classification from sum score cutoffs. We also discuss considerations for how to use scale scores in subsequent analyses and how these choices can alter conclusions. The general goal is to encourage researchers to more critically evaluate how they obtain, justify, and use multiple-item scale scores.

Psychometric Testing of the Chinese Version of the Coping and Adaptation Processing Scale-Short Form in Adults With Chronic Illness

“…Scores for three dimensions of the MCMQ were summed up separately. The summed scores were used within a latent variable framework (McNeish and Wolf, 2020). Further, the scores of item responses from 15-item CAPS-SF were summed up to achieve an approximation of adaptive capacity.…”

Toward Multimodal Modeling of Emotional Expressiveness

“…The weak negative correlation with neuroticism suggests that this context may attenuate displays of negative emotion (e.g., to make a good first impression). Given this evidence of the latent variable's internal consistency and external validity, we felt confident using the factor scores as labels of emotional expressiveness in our predictive modeling experiments [42].…”

“…After being averaged across raters, scores on the four expressivenessrelated questions were highly inter-correlated (all > 0.83), which supports the feasibility of combining these scores into a single lower-dimensional representation. The simplest way to combine the scores would have been to sum or average them, but to do so would assume that all questions are equally important and equally well-measured; these assumptions are unlikely to be met in practice and would negatively impact the validity and reliability of the aggregate if violated [42]. Thus, to avoid these assumptions, we used confirmatory factor analysis (CFA) [34] to estimate a latent variable that explains the variance shared among the questions (Figure 3).…”

“…In this model, factor loadings represent how much of each question was composed of shared variance (i.e., how strong the relationship was between the question and the latent variable), and residual variances represent how much each question was composed of non-shared variance (i.e., unique variance and measurement error). Because these model parameters are freely estimated (i.e., informed by the data), the latent variable will be a better representation of the shared variance than a sum or average would be [42].…”

The PAX Good Behavior Game: One Model for Evolving a More Nurturing Society

“…There were three different approaches to exploring subgroup effects based on baseline measurements, Latent Transition Analysis (LTA), the Johnson-Neyman technique, and cut-off scores. Both Johnson-Neyman and cut-off scores use rating scale sum scores, which can be problematic (McNeish & Wolf 2020 ), while LTA also takes the measurement model and measurement error into account. LTA is likely to be the most robust analysis method, but it requires large datasets to be appropriate.…”