Convergent and Discriminant Validation of Two Methods of Assessing Three Academic Traits

Abstract

A refined and extended analysis of variance (ANOVA) approach was used to investigate the pattern of relationships among arithmetic ability, reading ability, and general ability as assessed by teacher ratings and standardized tests. The subjects were 54 educable mentally retarded children who were rated and tested annually for 3 years.The tests proved to have much higher convergent and discriminant validity than the ratings, and to be much less affected by method bias. These differences are seen as reflecting the natural superiority of standardized tests over teacher ratings for most purposes.More than 20 years have passed since Campbell and Fiske (1959) introduced the notion of convergent and discriminant validation by the multitrait-multimethod (MTMM) matrix: a matrix containing the intercorrelations that result when each of several traits is measured by each of several methods. Essentially, their validity criteria were that measures of the same trait involving different methods should not only be substantially correlated, but that they should also correlate more highly than (a) measures of different traits involving different methods or (b) measures of different traits involving the same method. The first criterion is evidence of convergent validity, whereas the second and third provide evidence for discriminant validity. While noting that various statistical treatments for evaluating MTMM matrices might be developed, Campbell and Fiske chose to leave that challenging task to others. Nor have others been slow to take up the challenge; in a recent review of the literature, Schmitt, Coyle, and Saari (1977) illustrated six different approaches to the analysis of MTMM matrices and discussed their relative advantages and disadvantages.The approach employed in the present study is basically one proposed by Kavanagh, MacKinney, and Wolins (1971), following earlier work by Stanley (1961), Wolins (Note 1), and Zyzanski (1962). In this approach, the essential information in an MTMM matrix is expressed in terms of the usual mean squares of a three-way (subjects x traits x methods) factorial analysis of variance (ANOVA). Specifically, the main effect of subjects is taken to represent convergent validity;

Social Relationships and School Readiness

“…However, there are questions about the accuracy of teachers' judgments of children's skills and abilities. Teachers' judgments of children's readiness for school may be influenced by the goals or philosophies of the school or program within which they work, their implicit definitions of competencies that are needed for children to be ready for school, the performance levels of children in their classes, or other personal or psychological characteristics of teachers (Hoge, 1983;Hoge & Butcher, 1984;Mashburn & Henry, 2004;Silverstein, Brownlee, Legutki, & MacMillan, 1983). For example, Mashburn and Henry (2004) examined child, teacher, and program characteristics that were associated with prekindergarten teachers' ratings of children's readiness for kindergarten.…”

Validation of the Gifted Rating Scales–School Form in China

“…Some researchers have even voiced concerns about the validity and reliability of teacher nomination and ratings of gifted children (Hoge & Coladarci, 1989). Teachers' judgments on student performance are thought to be subjectively contaminated by their expectations and biases (Hoge, 1983;Hoge & Butcher, 1984;Sharpley & Edgar, 1986;Silverstein, Brownlee, Legutki, & MacMillan, 1983).Most authorities, however, argue that teachers have sufficient amounts of time to reliably observe and interact with students on a daily basis and that they are capable of making valid student assessments (Kenny & Chekaluk, 1993; Meisels, Bickel, Nicholson, Xue, & AtkinsBurnett, 2001; Perry & Meisels, 1996). Thus, teacher ratings and nomination are still considered very important data in the screening and identification of gifted students (Davidson, 1986;Pfeiffer, 2002).…”

Trusting Teachers’ Judgments: A Validity Study of a Curriculum-Embedded Performance Assessment in Kindergarten to Grade 3

“…Also questioned are teachers' abilities to discriminate such constructs as achievement and motivation and such individual differences as low achievement and specific learning disabilities (Hoge & Butcher, 1984;Salvesen & Undheim, 1994). Other areas of concern are the subjectivity inherent in teachers' judgments (Silverstein, Brownlee, Legutki, & MacMillan, 1983) and the extent to which teachers' expectations and biases may influence student outcomes (Hoge, 1983;Hoge & Butcher, 1984;Sharpley & Edgar, 1986). Given these concerns, and their implications for students, it is reasonable to ask, "Can we trust teachers' judgments of student performance?"…”