Evaluating Reading Diagnostic Tests: An Application of Confirmatory Factor Analysis To Multitrait-Multimethod Data

Marsh, Herbert W.; Butler, Susan R.

doi:10.1177/014662168400800308

Cited by 27 publications

(24 citation statements)

References 12 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Kenny (1994) stated that there are three basic types of methods that can be employed in the MTMM design: a) raters, b) instrument-based (different rating formats), and c) temporally-based methods (i.e., the same traits are measured on different occasions), While Becker and Cote (1994) provided by far the most comprehensive comparison of the three models, inspection of the studies included revealed that almost all matrices (14 of 17) used instrument-based methods, two (both from Marsh & Butler, 1984) used a mixture of method types, and only one (Schmitt, Coyle & Saari, 1977) used rater-based methods.…”

Section: Limitations Of Previous Researchmentioning

confidence: 99%

Analysis and Design of Multitrait-Multirater Performance Appraisal Studies

Conway

1996

Journal of Management

View full text Add to dashboard Cite

Becker and Cote (1994) found that the correlated uniqueness model outperformed the confirmatory factor analysis and direct product models for multitrait-multimethod data. The present study analyzed20 multitrait-multiraterperformance appraisal matrices. The correlated uniqueness model was appropriate significantly more often than in Becker and Cote5 study and the other two models performed poorly. The proportions of trait and method variance in ratings were related to several rating system characteristics such as opportunity for raters to observe ratees and basing rating dimensions on a job analysis. Performance of all three models was better with larger proportions of trait variance and smaller proportions of method variance.A good deal of research has addressed the analysis of multitrait-multimethod (MTMM) data. In a recent comprehensive review, Becker and Cote (1994) compared the performance of three models: the general confirmatory factor analysis model (CFA), the correlated uniqueness model (CU), and the direct product model (DP). It was found that the CU model produced the best results. However, Becker and Cote also found that all three models sometimes produced poor solutions (improper parameter estimates and/ or poor fit to the data). One purpose of the present study was to extend these results by applying the same three models to multitrait-multirater (MTMR) performance appraisal matrices (Becker and Cote's study included no such matrices). A second purpose was to go beyond previous work by linking analysis to substantive research design characteristics. In other words, this study investigated how researchers can design MTMR studies to produce high proportions of trait variance, low proportions of method variance, and to maximize the likelihood of good solutions. CFA, CU and DP ModelsSince Becker and Cote (1994) provided an extensive discussion of the three models and related evidence, brief (non-mathematical) descriptions are provided here.

show abstract

Section: Limitations Of Previous Researchmentioning

confidence: 99%

Analysis and Design of Multitrait-Multirater Performance Appraisal Studies

Conway

1996

Journal of Management

View full text Add to dashboard Cite

show abstract

“…Marsh and Butler (1984). using a version of the test designed for younger students called the GAP, reported that the test correlated .82 with the total score from the Stanford Diagnostic Reading Test (Karlsen.…”

Section: %mentioning

confidence: 99%

The outward bound bridging course for low-achieving high school males: Effect on academic achievement and multidimensional self-concepts

Marsh

Richards

1988

Australian Journal of Psychology

Self Cite

View full text Add to dashboard Cite

The Outward Bound Bridging Course is a six-& residential program daigned to improve academic achievement and selfconcepts in low-achieving high school males. In the period 1980-1984 five courses were conducted for 66 high school males chosen on the basis of poor academic performance, an ap arcnt potenfial to perfom better and strong parental su port. coupled with parental involvement as an academic intervention for low-achieving high school maks on both academic achievement and academic selfconce t; and (b) the validity of multidimensional selfconapt responses to the Self Dcscri tion &cstionnaire in relation to short multiple time-senes design, the specificity of the effects to academic outcomes and the generality of the effects across academk selfconapt and achievement make implausible many possible internal and external threats to the validity of the interpretations.The findings provide support P or: (a) the effective-of the Outward Bound Bridging ( ! o u m academic perfonnana and in relation to the impact of an e 8 &in academic intervention. The The Outward Bound Program Richards (1977) stated that the purpose of Outward Bound courses is to provide a setting for "the person to recognise and understand his own weaknesses, strengths and resources and thus find within himself the wherewithall to master the difficult and unfamiliar"(p. 69). The Outward Bound standard course is a 2May residential program for 17-25 year olds that comprises physically and mentally demanding outdoor activities. Marsh, Richards and Barnes (1986~ 1986b) demonstrated that participation in this standard course had a significant effect on the non-academic dimensions of selfconcept most related to the course goals and produced a more internal locus of control (see Ewart, 1983; Godfrey, 1974; Richards, 1977; Shore, 1977 for reviews of other Outward Bound march). The Bridging Course was subsequently developed by Outward Bound for low-achieving high school males. According to Richards and Richards (198 I): The authors would like to thank Jennifer Bane, Raymond Debus and Margarct Myen for their helpful cornmenu on earlier drafts of this paper, to acknowledge financial support from the Outward Bound Foundation; and to acknowledge the assistance of the Outward Bound R q u a t s for reprints should be sent to D r Herbert W. Marsh, Faculty of Education, University of Sydney, New South Wales 2006, Austnlia Comspondeacc specifically related to the Outward Bound program and the Bridging Course should be sent to Garry Richards, Director, Outward Bound Australin, P.O. Thaws, A.C.T. 2620, Australia Inquiries about the purchase or usc of the SDQ s e l f a n a p t instnUmnts should be sent to the Psycholopal Corporation, 555 Academic Court, San Antonio, Texas 282042498. USA. staff in conduaing this study. A&& J o d Of PIychdoy VO~. 40, NO. 3,1988 pp. 281-298

show abstract

“…Numerous examples of the application of CFA techniques to analyze MTMM matrices can be found in the psychological literature dating back more than 26 years (e.g., KM]eberg & Kluege], 1975; Kenny, 1976;Lance, Teachout, & DonneUy, 1992;Marsh & Butler, 1984;Marsh & Hecevar, 1985Rezmovic & Rezmovic, 1981;Scbmltt & Saari, 1978;Vance, MacCallum, Coovert, & Hedge, 1988;Wefts & Linn, 1970). Widaman (1985) notes that a researcher must first identify a model that adequately represents a set of data (i.e., the initial model) before pro-ceeding to estimate the convergent and discriminant validity by making comparisons of this initial model to models nested within it.…”

Section: Construct Validity and Confirmatory Factor Analysismentioning

confidence: 99%

Distinguishing personality and biodata items using confirmatory factor analysis of multitrait-multimethod matrices

Shultz

1996

J Bus Psychol

View full text Add to dashboard Cite

ABSTRACT'. Hard biodata, soft biodata, and personality items were developed to measure three personal constructs (Dependability, Demeanor, and Ambition). This resulted in nine sets of items forming a three-method (personality, soft biodata, and hard biodata) by three-trait (Dependability, Demeanor, and Ambition) Multitrait-Multimethod (MTMM) matrix. Four models were proposed to represent the nine scales and each was tested on a sample of 477 job applicants and subsequently cross-validated on a sample of 958 job applicants, using confirmatory factor analysis. The model where the personality and soft biodata items represented one factor and the hard biodata items represented another, distinct factor, was both the best fitting and most parsimonious model, thus confirming Asher's (1972) taxonomy.

show abstract

Evaluating Reading Diagnostic Tests: An Application of Confirmatory Factor Analysis To Multitrait-Multimethod Data

Cited by 27 publications

References 12 publications

Analysis and Design of Multitrait-Multirater Performance Appraisal Studies

Analysis and Design of Multitrait-Multirater Performance Appraisal Studies

The outward bound bridging course for low-achieving high school males: Effect on academic achievement and multidimensional self-concepts

Distinguishing personality and biodata items using confirmatory factor analysis of multitrait-multimethod matrices

Contact Info

Product

Resources

About