Less Truth Than Error: Massachusetts Teacher Tests

Abstract

Scores on the Massachusetts Teacher Tests of reading and writing are highly unreliable. The tests' margin of error is close to double to triple the range found on well-developed tests. A person retaking the MTT several times could have huge fluctuations in their scores even if their skill level did not change significantly. In fact, the 9 to 17 point margin of error calculated for the tests represents more than 10 percent of the grading scale (assumed to be 0 to 100). The large margin of error means there is both a high false-pass rate and a high false-failure rate. For example, a person who received a score of 72 on the writing test could have scored an 89 or a 55 simply because of the unreliability of the test. Since adults' reading and writing skills do not change a great deal over several months, this range of scores on the same test should not be possible. While this test is being touted as an accurate assessment of a person's fitness to be a teacher, one would expect the scores to accurately reflect a test-taker's verbal ability level. In addition to the large margin of error, the MTT contain questionable content that make them poor tools for measuring test-takers' reading and writing skills. The content and lack of correlation between the reading and writing scores reduces the meaningfulness, or validity, of the tests. The validity is affected not just by the content, but by a host of factors, such as the conditions under which tests were administered and how they were scored. Interviews with a small sample of test-takers confirmed published reports concerning problems with the content and administration.

The Myth of the Texas Miracle in Education

“…These test-retest correlations based on both the Linton-Debeic and Fassold data are, however, attenuated in that in both data sets only students who failed a TAAS test took it again. There are methods for correcting observed test-retest correlations for such attenuation (see Haney, Fowler and Wheelock, 1999, for an example), but as a more conservative approach here, let me simply discuss what previously published literature suggests about the relationships between test-retest and internal consistency reliability.…”

Legal and Psychometric Criteria for Evaluating Teacher Certification Tests

“…Fourth, the state testing agency and the test developer did not release any technical documentation regarding the psychometric quality of the test. Criticisms of the Massachusetts Teacher Test can be found in Haney, Fowler, Wheelock, Bebell, and Malec (1998). It is likely that problems with this test administration could be found from both legal and psychometric perspectives.…”

There is another way: The faculty-developed Idaho Comprehensive Literacy Assessment for K-8 pre-service teachers

“…Teacher testing in Massachusetts received considerable attention. The highly politicized and publicized teacher testing resulted in firestorm of controversy (Flippo, & Riccards, 2000;Haney, Fowler, & Wheelock, 1999;Ludlow, Shirley, and Rosca, 2002). Apparently, the content of the test is unconnected to good teaching or what teachers actually do (Ludlow, Shirley, & Rosca).…”

“…Apparently, the content of the test is unconnected to good teaching or what teachers actually do (Ludlow, Shirley, & Rosca). In addition, although the Massachusetts test is lauded as a mastery test, norm-referenced test development procedures were used, undermining the mastery intent of the test (Haney, 1999). The end result is the mis-measurement of teacher competence in Massachusetts.…”