Everything that you have ever been told about assessment center ratings is confounded.

Abstract

Despite a substantial research literature on the influence of dimensions and exercises in assessment centers (ACs), the relative impact of these two sources of variance continues to raise uncertainties because of confounding. With confounded effects, it is not possible to establish the degree to which any one effect, including those related to exercises and dimensions, influences AC ratings. In the current study (N = 698) we used Bayesian generalizability theory to unconfound all of the possible effects contributing to variance in AC ratings. Our results show that ≤ 1.11% of the variance in AC ratings was directly attributable to behavioral dimensions, suggesting that dimension-related effects have no practical impact on the reliability of ACs.Even when taking aggregation level into consideration, effects related to general performance and exercises accounted for almost all of the reliable variance in AC ratings. The implications of these findings for recent dimension-and exercise-based perspectives on ACs are discussed. When, in the context of selection, appraisal, and development, behavioral criteria are used to evaluate individuals, it is essential that these criteria are measured reliably.Unsurprisingly, therefore, the measurement properties of assessment center (AC) ratings have come under close scrutiny in the applied psychology literature. In ACs, the behavior of jobholders or candidates is sampled across several work-related situations (exercises, e.g., a role play exercise, group discussion, presentation) and is typically assessed by trained assessors in terms of pre-defined behavioral dimensions (e.g., communication skills, teamwork, planning and organizing). As a result of their multifaceted measurement properties, incorporating dimensions, exercises, and assessors, ACs provide a rich source of information about the extent to which work-related behavioral criteria can be reliably measured in a job-relevant setting.Historically, researchers have questioned the extent to which behavioral dimensions are measured reliably in ACs, and have implied that researchers should utilize an exercise-oriented approach to scoring ACs

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

“…SJT was developed for an educational context, which might have led to findings that are specific to educational SJTs. However, the pervasive exercise effects observed as a matter of routine in the context of AC ratings (see Jackson et al ., ) bear similarities to the findings reported by Westring et al . in the context of SJTs.…”

“…While the findings of Krumm et al are seemingly at odds with those of Westring et al (2009), the Westring et al SJT was developed for an educational context, which might have led to findings that are specific to educational SJTs. However, the pervasive exercise effects observed as a matter of routine in the context of AC ratings (see Jackson et al, 2016) bear similarities to the findings reported by Westring et al in the context of SJTs. Thus, it seems that further investigation is warranted into this key feature of the SJT design.…”

“…() as well as to our findings. Note that it has been argued elsewhere that ‐analogous effects represent a type of situational effect (Jackson et al ., ).…”

“…In particular, the assessment centre (AC) literature has grappled with analogous dimensions and their contribution to AC ratings for over six decades (see Sakoda, 1952). While the AC context is different from that presented by SJTs, the AC literature has, nonetheless, utilized innovative analytic approaches to help inform on complex psychometric designs (e.g., Jackson, Michaelides, Dewberry, & Kim, 2016;Woehr, Meriac, & Bowler, 2012;Woehr, Putka, & Bowler, 2012).…”

“…Putka and Hoffman () and Jackson et al . () recently applied this approach to ACs and demonstrated that G theory can be used to summarize nuanced and informative components of reliable (i.e., true score‐analogous) and unreliable (i.e., error‐analogous) variance in an organizational multifaceted measure.…”