Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis.

Abstract

We used hierarchical confirmatory factor analysis to test alternative latent structures underlying performance in multiple dimensions as measured by work samples and three rating sources. The model receiving strongest support was one that specified multidimensional trait (performance dimension) and measurement method second-order factors (SOFs). This model exhibited significant convergent and discriminant validity, showed significant measurement method effects, and indicated that different measurement methods assess somewhat different aspects of the total criterion construct space. Significant correlations between measurement method SOFs and performance predictors suggest the need to reconsider the interpretation of method factors, not as bias, but as possibly representing different overall perspectives on performance.

Decision-Making Styles and Subjective Performance Evaluation of Decision-Making Quality among Hospital Nurses

“…Conversely, Lance et al (2008) argue from an ecological perspective that "rater source effects do not represent (mere) rater biases but rather represent alternative but complementary valid perspectives on ratee performance" (p. 227). Multiple studies support their view by detecting strong relationships between rater source factors and performance-based external variables (Lance, Baxter, & Mahan, 2006;Hoffman & Woehr, 2009;Lance, Teachout, & Donnelly, 1992). Thus, ratings from different sources can be considered as rather valid measures of performance, even though their applicability across different conditions certainly deserves further study.…”

Method Variance and Method Bias in Industrial and Organizational Psychology

“…It is important to note the argument by Lance, Teachout, and Donnelly (1992) regarding method variance in performance ratings. Lance et al cautioned against assuming that rater variance is a method contaminant, presenting evidence that variance unique to a rater contains interpretable, substantive variance.…”

Distinguishing personality and biodata items using confirmatory factor analysis of multitrait-multimethod matrices

“…Numerous examples of the application of CFA techniques to analyze MTMM matrices can be found in the psychological literature dating back more than 26 years (e.g., KM]eberg & Kluege], 1975; Kenny, 1976;Lance, Teachout, & DonneUy, 1992;Marsh & Butler, 1984;Marsh & Hecevar, 1985Rezmovic & Rezmovic, 1981;Scbmltt & Saari, 1978;Vance, MacCallum, Coovert, & Hedge, 1988;Wefts & Linn, 1970). Widaman (1985) notes that a researcher must first identify a model that adequately represents a set of data (i.e., the initial model) before pro-ceeding to estimate the convergent and discriminant validity by making comparisons of this initial model to models nested within it.…”

“…Therefore, it is proposed that by obtaining a better understanding of the methods used, before analyzing the constructs that they purport to represent, researchers will be able to have a better understanding of the methods as well as a clearer interpretation of the constructs those methods are purported to measure. Lance, et al (1992) present an excellent example of interpreting method factors, not as bias, but as representing differing aspects of a larger criterion construct space.…”