The Usefulness of Unit Weights in Creating Composite Scores

Abstract

Combining/weighting subscores into an aggregate score involves issues that apply to many fields in the organizational sciences (e.g., weighting predictors in selection, weighting multiple performance appraisal indicators, overall evaluation of organizations). The weights that are used in practice can be different (differential weights) or equal (unit weights). Relevant literature across multiple disciplines and multiple decades is reviewed. The literature indicates that unit weights have substantial predictive validity when compared with regression weights, but there is a lack of data on how other differential weighting strategies (e.g., weights generated by subject matter experts) compare to unit weights. In response, a primary and a meta-analytic study are provided here. The recent literature also contains some potential criticisms of unit weights in regard to personnel selection and content validation—and those statements are evaluated. The data and findings indicate that unit weights can be a highly appropriate approach for weighting under many circumstances.

Medication adherence for resistant hypertension: Assessing theoretical predictors of adherence using direct and indirect adherence measures

“…First, it provided an opportunity to compare adherence scores as measured by drug levels in urine, self-report, and prescription refill records within a large sample of aTRH patients in primary care. Second, this study used rigorous and systematic statistical methods informed by psychometric theory (Bobko et al, 2007) to create a composite adherence score for a large sample of patients that can be replicated for future cohorts. Third, our replication of previous empirical findings using an enhanced methodology of adherence assessment by including a physical measure of patient adherence in the form of urine assays provides additional evidence for the key role of habit strength in predicting long-term adherence behaviour.…”

“…Adherence composite. As each adherence measure has established construct validity but is also subject to varying kinds of measurement biases (Osterberg & Blaschke, 2005), a unit-weighted composite adherence score was calculated by standardising and summing scores from each individual adherence measure (Bobko, Roth, & Buster, 2007). The use of multiple methods to assess medication-taking behaviour is recommended in this literature due to the inherent limitations of each individual measure (Bond, 2016).…”

Validity and Adverse Impact Potential of Predictor Composite Formation

“…Obviously, using a more balanced Pareto‐optimal composite will lead to a somewhat different selected workforce than the one obtained when using the regression‐based composite. The difference between the selected workforces will usually be small, however, as the more balanced composites typically have a high to very high correlation with their corresponding regression‐weighted composite (e.g., Bobko et al , 2007). Thus, for the presently considered predictors we found that all of the 90% or higher maximum validity composites showed at least a .85 correlation with their corresponding regression‐based composite.…”

Some Comments on Pareto Thinking, Test Validity, and Adverse Impact: When ‘and’ is optimal and ‘or’ is a trade‐off

“…An important line of research related to understanding selection test composites, to which De Corte et al contribute, concerns how organizations might weight tests within the composite. As noted above, some researchers used unit‐weights (e.g., Sackett & Ellingson, 1997 or, more recently, Bobko, Roth, & Buster, 2007). Yet other researchers have used multiple regression weighting to analyze AI potential at various selection ratios.…”