Three method factors explaining the low correlations between assessment center dimension ratings and scores on personality inventories

Abstract

In general, correlations between assessment centre (AC) ratings and personality inventories are low. In this paper, we examine three method factors that may be responsible for these low correlations: differences in (i) rating source (other versus self), (ii) rating domain (general versus specific), and (iii) rating format (multi-versus single item). This study tests whether these three factors diminish correlations between AC exercise ratings and external indicators of similar dimensions. Ratings of personality and performance were combined in an analytical framework following a 2 Â 2 Â 2 (source, domain, format) completely crossed, within subjects design. Results showed partial support for the influence of each of the three method factors. Implications for future research are discussed. Copyright # 2004 John Wiley & Sons, Ltd. It is well known that assessment centres (ACs) measure job relevant constructs, thanks to a satisfactory criterion-related validity (Gaugler, Rosenthal, Thornton, & Bentson, 1987). Unfortunately, we do not know exactly what these constructs are (Russell & Domm, 1995). This question has resulted in an abundant body of research seeking the meaning of the AC dimensions. These studies have mainly focused on the effects of the AC architecture (dimensions, exercises, assessor characteristics, etc) on construct validity (see Lievens and Conway, 2001, for a review). Evaluating the results of two decades of research, Lance et al. (2000, p. 344) noted that 'In retrospect, we think the question ''we know what the assessment center constructs are (i.e., dimensions); are these valid?'' was premature '. Lance et al. proposed that Russell's (1994) question, 'what are the assessment center constructs?', needed to be answered first. To answer the latter question, constructrelated validity of ACs has been studied by placing AC dimension ratings in a nomological network of cognitive ability tests and personality inventories. While some of the results using the external construct validity approach were actually promising (Scholz & Schuler, 1993;Shore, Shore, & Thornton, 1992;Shore, Thornton, & Shore, 1990), other studies failed to show the expected relationships between AC dimension ratings and these external measures. Specifically, it appeared that cognitive ability showed some relatedness with AC ratings, while the AC-personality inventory correlation was often negligible (see e.g. Borman, 1982;Bray & Grant, 1966;Chan, 1996;Crawley, Pinder, & Herriot, 1990;Fleenor, 1996;Hinrichs, 1978;Lance et al., 2000;Tziner & Dolan, 1982). This latter finding appears to be yet another in a long line of evidence against the AC dimension's construct validity, adding concerns about external validity to a larger literature on internal construct validation problems (see e.g. Chan, 1996;Lievens & Conway, 2001;Sackett & Dreher, 1982). For instance, Crawley et al. (1990, p. 215) note that 'The general implication for the assessment centre design is to cast further doubt on the use of the ''sign'' as opposed to th...

Fitting a Mixture Item Response Theory Model to Personality Questionnaire Data: Characterizing Latent Classes and Investigating Possibilities for Improving Prediction

“…In personality measurement, as in the present study, correlations with external criteria are usually not very high. Moreover, it has been shown that correlations between self-ratings and ratings by others, such as a psychologist, are lower than self-self or other-other ratings (Kolk, Born, & van der Flier, 2004). Overall, mixture IRT models provide possibilities to improve the prediction of external criteria, but it may vary across personality scales.…”

Personality predispositions and competence

“…The results of Kolk research (Kolk, Born, & Flier, 2004) throw an interesting light on this matter. In this research attention was paid to the fact that personality questionnaires are the result of self-assessment, which is carried out by the person tested, while in the case of AC results, assessment is conducted by external observers and, what is more, it is a time limited assessment.…”

Assessor Constructs in Use as the Missing Component in Validation of Assessment Center Dimensions: A critique and directions for research

“…In a similar vein, Kolk, Born, and Van der Flier (2004) and Bartram (2005) rely on a content ‐based approach, meaning that dimension definitions are the particular focus of their research. Kolk et al (2004) devised some center dimensions based on a cogent analysis of the applied personality and social cognition literature. This deductive approach may serve several functions, not least of which is to rely on existing KSAO labels.…”