An Examination of the Effects of Using Behavior Checklists on the Construct Validity of Assessment Center Dimensions

Abstract

Although research has established the criterion-related validity of assessment centers for selection purposes, the construct validity of dimension ratings has not been demonstrated. A quasi-experimental design was used to investigate the influence of retranslated behavior checklists on the construct validity of dimension ratings for two assessment center exercises. Assessor use of behavior checklists increased the average convergent (i.e., same dimension across exercise) validity from .24 to .43 while decreasing the average discriminant (i.e., different dimension within exercise) validity (.47 to .41). Behavior checklist sums were moderately correlated with corresponding dimension ratings and demonstrated a comparable level of construct validity. It is suggested that using behavior checklists may improve dimension construct validity by reducing the cognitive demands placed on raters.

Assessment Centre Observation Procedures: An Experimental Comparison of Traditional, Checklist and Coding Methods

“…One study is relevant to the issues raised here. Reilly et al (1990) found that the average betweendimension correlation within exercises was similar in Traditional and Behavioural Checklist conditions, contrary to their expectation that the latter procedure would yield lower intercorrelations because of reduced cognitive load and the provision of more detail about dimensional content. Additional evidence about between-dimension correlations would be welcome, and comparisons will be made between the three observational methods examined here.…”

Constructs and developmental assessment centers: Further troubling empirical findings

“…The use of design interventions, such as scoring checklists, may reduce the cognitive demands on assessors (Reilly, Henry & Smither, 1990). Silverman et al (1986) described a within-dimension method of producing AC ratings, which resulted in some improvement in their construct validity.…”

Are the Communication and Professionalism Competencies the New Critical Values in a Resident’s Global Evaluation Process?

“…These results are consistent with a substantial body of research on rater errors, including "halo error" (when salient aspects of a resident's performance on 1 competency bias the evaluator's judgments of the resident's performance on other, unrelated competencies), "central tendency error" (the tendency to rate all residents as being "average" performers), and the "recency effect" (when recent examples of the resident's behavior biases the evaluator's recall of the resident's earlier behaviors). 18 The effects of rater errors have been known for over 60 years, when assessment centers were first used by the Office of Strategic Services (OSS) to select candidates for clandestine missions in occupied Europe. 19 Similar findings have been observed in other domains, such as assessing pilot crew performance in commercial aviation 20,21 and assessing management potential in the business community.…”