Issues in managerial performance: Multitrait-multimethod analyses of ratings.

Abstract

Three general issues of dimensionality, validity, and content of ratings of managerial performance are examined in terms of "relevance to the ultimate criterion." It is argued that the multitrait-multimethod design yields the best evidence for investigating these issues. An analysis of variance model with computational formulas for the sums of squares and variance components based on the correlation matrix is proposed for the multitrait-multimethod situation. Using this model, data from two studies are examined and compared, relative to convergent and discriminant validity, method bias (halo), and error variance. It appears that this model and the indexes derived from it provide a more simplified and interpretable technique for analyzing and summarizing multitrait-multimethod data. In addition, a procedure to improve performance ratings based on partitioning the original matrix is illustrated with an example from the larger data set.

Bias in the Correlated Uniqueness Model for MTMM Data

“…Although Campbell and Fiske's article broke new and important methodological ground, it has been pointed out that their criteria for establishing convergent and discriminant validity were qualitative and subjective (e.g., Schmitt & Stults, 1986). As a result, numerous, more quantitatively based approaches to the MTMM matrix have been proposed, including (a) analysis of variance (ANOVA; e.g., Kavanagh, MacKinney, & Wolins, 1971), (b) path analysis (e.g., Kalleberg & Kluegel, 1975), (c) linear regression (e.g., Lehman, 1988), (d) exploratory factor analysis (e.g., Lomax & Algina, 1979), (e) covariance components analysis (e.g., Wothke, 1996), and (f) confirmatory factor analysis (CFA; e.g., Widaman, 1985).…”

Construct Validity of Leader Effectiveness Criteria

“…The third step in the analysis was based on the analysis of vanance model described by Kavanagh et al (1971). In brief , the model is a three-way factorial design from which four sources of variance can be derived to capture the essence of a multitrait-multimethod matrix .…”

“…• source variance , indicating method/source bias; and (d) error ( Kavanagh • • et al, 1971). In addition , the amount of variance accounted for by each source can be estimated to facilitate relative comparisons anx ng the variance components; this is not feasible in an analysis restricted solely to the correlations in a multitrait-multimethod matrix .…”

“…The Kavanagh et al (1971) study is particularly interesting because it suggests an analysis of variance technique that simplifies the analysis and interpretation of a multitrait-multimethod matrix .…”

Sociological and economic factors related to managerial success

“…Another way to evaluate managerial effectiveness is to select an objective criteria such as number of promotions, salary increases, or profits (Hulin, 1963, Kavanagh et al, 1971. In a large corporate structure, promotions or salary increases may serve quite well as indicators of an individual's managerial skills.…”