A Multitrait-Multimethod Validity Investigation of Scores from a Professional Licensure Examination

Abstract

The construct validity of scores from the Uniform CPA Examination was investigated through the construction of a multitrait-multimethod matrix. The four sections of the exam were treated as distinct traits, and within-section subscores were created according to item format. First, Campbell and Fiske’s four criteria were used to evaluate the matrix. After correcting the correlations for attenuation due to unreliability, evidence was moderate for the presence of four discrete constructs within the exam. The monotrait-heteromethod correlations were large and statistically significant and in a majority of cases were larger than other correlations found within the matrix. Structural equation models were fit to the data to explore whether excessive method variance was present. The results indicated that there was very little method variance present in the data. The implications for understanding the constructs measured by this exam, and for evaluating the current practice of reporting separate exam section scores, are discussed.

Evaluating the Measurement Properties of Procedural Justice in a Correctional Setting

“…In this instance, the mode of administration is the same but by the question format represents distinct methods. A recent MTMM utilized different question formats on certified public accountant (CPA) examinations to measure similar and distinct concepts (Pitoniak, Sireci, & Luecht, 2002).…”

Predicting assessment center performance with 360‐degree, top‐down, and customer‐based competency assessments

“…Correlations for the same competency (e.g., Oral Presentation) measured with different methods (e.g., 360 versus AC) should be significantly different than zero and large enough to encourage additional examination of validity (D. T. Campbell & Fiske, 1959). In addition, each convergent validity correlation should be larger than the other correlations found in its row and column that measure different competencies by different methods and different competencies measured with the same method (Pitoniak, Sireci, & Leucht, 2002).…”

Statistical validation of air-displacement plethysmography for body composition assessment in children

“…Alternatively, the multitrait-multimethod matrix (MTMM) is a correlation-based framework for comparing the reliability and validity of techniques purported to measure similar traits using multiple methods of assessment (Campbell and Fiske 1959). MTMM has been used for decades to develop and validate measures in the fields of education (Bong and Hocevar 2002;Pitoniak et al 2002), health policy management (Hadorn and Hays 1991;Krabbe et al 1997;Wann-Hansson et al 2004), human factors engineering (Brannick et al 1995;Schroeder et al 2002;Ketokivi and Schroder 2004), nursing (Ferketich et al 1991), and psychiatry (Bastiaansen et al 2004). In addition, factor analytic methods are considered by many to be an essential tool in the instrument validation process.…”

“…In addition, factor analytic methods are considered by many to be an essential tool in the instrument validation process. They are particularly useful with respect to identifying the number of underlying dimensions in a data set and complement nicely the MTMM approach (Pitoniak et al 2002;DeVillis 2003).…”