Assessment Centre Observation Procedures: An Experimental Comparison of Traditional, Checklist and Coding Methods

Abstract

Concern about the construct validity of assessment centre judgements has led to calls for research into observers' cognitive processes. In an experiment comparing the Traditional observation procedure against use of a Behavioural Checklist and of Behavioural Coding, six outcome variables were examined. The methods were found to yield similar outcomes in terms of accuracy of judgement, accuracy of written evidence, correlation between dimension ratings, and attitude toward the method employed. However, significant between-method differences were observed in variability between observers and in their willingness to recommend a procedure. It is argued that Behavioural Coding has special merit in assessment centres; the method reduces the cognitive demands placed on assessors, and valuably structures their information-processing. Traditional objections to simultaneous observation and classification are shown to be inappropriate.

Assessment Center Procedures: Cognitive Load During the Observation Phase

“…Although the present results and the results by Hennessey et al (1998) do not support that taking notes increases the quality of the ratings, it cannot be uniformly concluded that assessors might as well skip taking notes during the exercise. The Guidelines (Task Force on Assessment Center Guidelines 1989) specifically state that a systematic procedure must be used by assessors to record accurately specific behavioural observations at the time of their occurrence.…”

“…However, results showed no significant differences between observation methods or types of assessors. This result is similar to Hennessey et al (1998) who also failed to find decreased inter-dimension correlation when they compared the traditional observation method with the behavioural checklists and coding method (this latter method involved tallying behaviours during the exercise on a behavioural checklist, rather than recording them). Yet, it is important to realize that both this study and the Hennessey et al study applied a design divergent from the regular candidate  dimension designs to study halo, because this design examines the ratings of multiple assessors and only a few candidates.…”

“…Halo refers to the phenomenon that a rater's general impression of candidates distorts their perception on specific dimensions. Following Hennessey et al (1998), we also examined correlations between dimensions within exercises for the two methods and types of assessors. These correlations can be regarded as measures of halo, although their meaning differs from those in studies using a regular assessee  dimension design, instead of an assessor  dimension design.…”

“…This dual task may require more cognitive processing capacity than assessors have available, potentially leading to more rating errors. Also, the assessors' attention has to shift from the exercise to their record-sheet, so that key behavioural information might be overlooked (Hennessey, Mabey and Warr 1998). Taking notes during observation may therefore be a mixed blessing.…”

A Comparison of Assessment Center Practices in Organizations in German‐speaking Regions and the United States

“…Reilly et al (1990) showed that the use of behavioral checklists or realistic behavioral descriptions improved construct validity because the assessors were able to relate assessee behavior more clearly to the various dimensions of job requirements. Hennessy, Mabey, and Warr (1998) experimentally demonstrated the superiority of behavioral checklists and behavioral coding over traditional observational systems. Behavioral observation scales usually only use key words to express the job requirements, which are not clearly enough delineated to begin with.…”