Constructs and developmental assessment centers: Further troubling empirical findings

Abstract

ABSTRACT:. Research indicates that assessment center (AC) ratings typically demonstrate poor construct validity; that is, they do not measure the intended dimensions of managerial performance (e.g., Sackett & Harris, 1988). The purpose of this study was to investigate the construct validity of dimension ratings from a developmental assessment center (N= 102), using multitrait-multimethod analysis and factor analysis. The relationships between AC ratings, job performance ratings, and personality measures also were investigated. Results indicate that the AC ratings failed to demonstrate construct validity. The ratings did not show the expected relationships with the job performance and personality measures. Additionally, the factors underlying these ratings were found to be the AC exercises, rather than the managerial dimensions as expected. Potentially, this lack of construct validity of the dimension ratings is a serious problem for a developmental assessment center. There is little evidence that the managerial weaknesses identified by the AC are the dimensions that actually need to be improved on the job. Methods are discussed for improving the construct validity of AC ratings, for example, by decreasing the cognitive demands on the assessors.

The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous Examinations of Construct-Related and Criterion-Related Validity

“…In one group of studies, AC ratings were linked to external criteria in a correlational design (Chan, 1996;Fleenor, 1996;Henderson, Anderson, & Rick, 1995;Jansen & Stoop, 2001). For example, Chan validated an AC for police officers in Singapore (N = 46).…”

A Reevaluation of Assessment Center Construct-Related Validity

“…Much of this research has been less than promising and tends to diminish the effects of dimensions on the variance of PEDRs (e.g., Lance, Lambert et al, 2004). Primarily utilizing CFA techniques, research has shown that models with large exercise effects tend to fit PEDR data better than those with large dimension effects (e.g., Bycio et al, 1987;Fleenor, 1996;Lance, Foster, et al, 2004;Lance, Lambert, et al, 2004;Lance et al, 2000;Schneider & Schmidt, 1992). The present study sought to challenge these findings by utilizing a relatively overlooked technique, univariategeneralizability theory, to examine the sources of variance in assessment center PEDRs.…”

“…That is, regardless of the true nature of the data, the 1DCE model is most likely to emerge as the appropriate model. Thus, the conclusions of the past 20 years of research regarding the construct-related validity of assessment centers, and more specifically, the results of numerous studies that have concluded that assessment center PEDRs are best represented by the 1DCE model (e.g., Bycio et al, 1987;Fleenor, 1996;Lance, Foster, et al, 2004;Lance, Lambert, et al, 2004;Lance et al, 2000;Schneider & Schmidt, 1992), may include some erroneous results.…”

Developing a framework for assessing effective development activities

“…Candidates who perceived to have better managerial support were also more likely to receive higher DC ratings; but we note that this study entailed retrospective baseline measures, and a small sample which taken together question the robustness of findings. Fleenor (1996) found that DC ratings bore no relation to a measure of managerial performance (the averaged ratings from a 360 degree feedback tool) concluding that the evidence showed that resulting developmental feedback may therefore be misleading and detrimental. However, individual career motivation was associated with taking development action and advancement and developmental recommendations only tended to be followed if rating feedback sent a positive message about future advancement; showing that feedback content and individual differences may interact.…”