Adjustment in assessment scores and their usage: A taxonomy and evaluation methods.

Abstract

Methods of adjusting group differences in assessment and test scores are described, classified, and evaluated. The adjustments are classified on the basis of (a) the type of adjusted difference, (b) the method of adjustment, and (c) the purpose of the adjustment. Adjustments used to improve the social fairness of outcomes either could not be fully evaluated or were inconsistent with the underlying fairness model. Adjustments designed to reduce score bias, in the technical sense, could be meaningful and internally consistent only if the observed group difference could be related to constructs targeted by the assessment procedure.

Commentary on Pareto‐Optimality as a Rationale for Adverse Impact Reduction: What would organizations do?

“…Their contribution to this research stream is to suggest that Pareto-optimality is a standard by which some amount of validity loss might be judged acceptable to organizations even if no combination exceeds the AIR legal threshold. This perspective is not unique and is highly similar to previous analyses of the trade-off between selection diversity and quality underlying various banding methods (e.g., Murphy, Osten, & Myors, 1995;Sackett & Roth, 1991;Kehoe & Tenopyr, 1994;Laczo & Sackett, 2004). As the authors acknowledge, the critical organizational decision is to be willing to accept a loss in selection quality in return for AIR improvement.…”

“…D e Corte, Lievens, and Sackett (2008) demonstrate that the component weights of a selection composite may be used to improve adverse impact ratios (AIR) more than previously described by considering Pareto-optimal composites rather than just optimal regression composites. This analysis is insightful and informative and extends a 15-year-old stream of personnel selection research on factors that affect AIR or, more generally, group differences on selection criteria and in selection decisions (Cascio, Outtz, Zedeck, & Goldstein, 1991;Sackett & Roth, 1991;Kehoe & Tenopyr, 1994;Murphy, Osten, & Myors, 1995;Schmitt, Rogers, Chan, Sheppard, & Jennings, 1997;Sackett & Ellingson, 1997;Hoffman & Thornton, 1997;Hough, Oswald, & Ployhart, 2001;Hattrup & Rock, 2002;Potosky, Bobko, & Roth, 2005;Roth, Bobko, & Switzer, 2006;De Corte, Lievens, & Sackett, 2006;De Corte, Lievens, & Sackett, 2007). The purpose of this commentary is to describe the organizational context in which Pareto-optimal solutions may be implemented and to evaluate whether such solutions are likely to be attractive to organizations.…”

Personnel Selection with Top‐Score‐Referenced Banding: On the Inappropriateness of Current Procedures

“…As noted earlier, there are many other technical inconsistencies with statistically based banding that have been pointed out (e.g., Kehoe & Tenopyr, 1994;Schmidt & Hunter, 1995). We briefly add two other logical problems we have noticed in our recent experience.…”

“…He indicates that applying the procedure to all values in the test score distribution implies that all tested applicants will be considered equal to all others. Also, Kehoe and Tenopyr (1994) noted that banding's hypothesis about ranges of statistical indifference are logically counter to the use of a non-zero reliability coefficient (this latter fact implying that true score variance must be greater than zero). Kehoe and Tenopyr also demonstrate that test score differences within the same band would indeed be likely to occur in the same direction on repeated assessments.…”

The Controversy Over Score Banding in Personnel Selection: Answers to 10 Key Questions

“…There have been many follow-up articles on the topic in Human Pegonnance since the Cascio et al (1991) article (Aguinis, Cortina, & Goldberg, 1998;Sackett & Roth, 1991;Schmidt, 1991;Schmidt & Hunter, 1995;Siskin, 1995;. There have also been articles on the topic in the American Psychologist Gottfredson, 1994;Sackett & Wilk, 1994), Personnel Psychology (Murphy, 1994;Murphy, Osten, & Myors, 1995), the Journal ofApplied Psychology (Truxillo & Bauer, 1999), and Psychologicalhsessment (Kehoe & Tenopyr, 1994), as well as a report on banding by the scientific affairs committee of the Society for Industrial and Organizational Psychology (Scientific Affairs Committee, 1994). The issues cover the entire spectrum from statistical and scientific, to practical and legal, to moral and ethical.…”