Analysis and Design of Multitrait-Multirater Performance Appraisal Studies

Abstract

Becker and Cote (1994) found that the correlated uniqueness model outperformed the confirmatory factor analysis and direct product models for multitrait-multimethod data. The present study analyzed20 multitrait-multiraterperformance appraisal matrices. The correlated uniqueness model was appropriate significantly more often than in Becker and Cote5 study and the other two models performed poorly. The proportions of trait and method variance in ratings were related to several rating system characteristics such as opportunity for raters to observe ratees and basing rating dimensions on a job analysis. Performance of all three models was better with larger proportions of trait variance and smaller proportions of method variance.A good deal of research has addressed the analysis of multitrait-multimethod (MTMM) data. In a recent comprehensive review, Becker and Cote (1994) compared the performance of three models: the general confirmatory factor analysis model (CFA), the correlated uniqueness model (CU), and the direct product model (DP). It was found that the CU model produced the best results. However, Becker and Cote also found that all three models sometimes produced poor solutions (improper parameter estimates and/ or poor fit to the data). One purpose of the present study was to extend these results by applying the same three models to multitrait-multirater (MTMR) performance appraisal matrices (Becker and Cote's study included no such matrices). A second purpose was to go beyond previous work by linking analysis to substantive research design characteristics. In other words, this study investigated how researchers can design MTMR studies to produce high proportions of trait variance, low proportions of method variance, and to maximize the likelihood of good solutions. CFA, CU and DP ModelsSince Becker and Cote (1994) provided an extensive discussion of the three models and related evidence, brief (non-mathematical) descriptions are provided here.

Transparency in Structured Interviews: Consequences for Construct and Criterion-Related Validity

“…Unlike the CTCM approach, the CTCU approach does not require separate method factors but rather infers method effects from correlated uniquenesses within methods (i.e., correlations among error terms of performance measures within the same method). Although the CTCU approach rarely produces ill-defined solutions, it has its own methodological and theoretical disadvantages (see Lance, Noble, & Scullen, 2002;Tomás et al, 2000), which have caused researchers to suggest the use of both CTCM and CTCU approaches when testing for construct validity (Conway, 1996;Lance et al, 2002).…”

Bias in the Correlated Uniqueness Model for MTMM Data

“…MTMM studies were defined broadly (Kenny, 1995), to include instrument-based, rater-based (so-called multitrait-multirater matrices), and temporally based methods (so-called multitrait-multioccasion matrices). Five large-scale reviews were identified and reviewed (Buckley, Cote, & Comstock, 1990;Conway, 1996;Hernandez & Gonzalez-Roma, 2002;Lievens & Conway, 2001;Williams, Cote, & Buckley, 1989). The Becker and Cote (1994) review was not included because the studies overlapped considerably with other reviews.…”

“…Two of the issues discussed by Lance et al are most critical for our study. The first issue is that the CU model returns convergent and proper solutions far more often than does the CTCM model (Conway, 1996;Marsh, 1990;Marsh & Bailey, 1991;Marsh, Byrne, & Craven, 1992). Because of this advantage, the CU model has gained in popularity relative to the CTCM model.…”

Intelligence, “Big Five” personality traits, and work drive as predictors of course grade

“…Task performance consists of technical proficiency at a skill considered important in a particular job. Contextual performance refers to activities outside a specific job description that support or promote the interests and goals of the company, such as working well with others and taking on tasks and responsibilities that are not assigned (Arvey & Murphy, 1998;Conway, 1996;Schmitt & Borman, 1993).…”

“…In addition to these measures, objective and self-report measures of employee absences, accidents, counterproductive behaviors, and specific task performance are used (Dunn, Mount, Barrick, & Ones, 1995;Johns, 1994;Lysaker, Bell, Kaplan, & G., 1998;Salgado, 2002). Conway (1996), in a review of performance appraisal studies, found support for the use of these two distinct performance categories, task performance and contextual factors, especially for non-managerial jobs. He did find, however, substantial inter-correlations between the two domains.…”