A Validity Argument Approach to Evaluating Teacher Value-Added Scores

Abstract

Value-added models have become popular in research and pay-forperformance plans. While scholars have focused attention on some aspects of their validity (e.g., scoring procedures), others have received less scrutiny. This article focuses on the extent to which value-added scores correspond to other indicators of teacher and teaching quality. The authors compared 24 middle school mathematics teachers' value-added scores, derived from a large (N = 222) district data set, to survey-and observation-based indicators of teacher quality, instruction, and student characteristics. This analysis found teachers' value-added scores correlated not only with their mathematical knowledge and quality of instruction but also with the population of students they teach. Case studies illustrate problems that might arise in using value-added scores in pay-for-performance plans.

Sorting out the signal: Do multiple measures of teachers’ effectiveness provide consistent information to teachers and principals?

“…Using samples that varied in size from just 24 teachers (Grossman et al, 2013;Hill et al, 2011) to over 3,000 teachers in seven districts across the country (the Measures of Effective Teaching [MET] project) (Kane & Staiger, 2012;Mihaly et al, 2013), these studies tested multiple observational instruments and relied on expert reviewers to observe live and videotaped lessons. Simple and disattenuated correlations between the scores from the instructional standards of observation-based measures and "underlying" value-added measures of teacher effectiveness ranged in the MET study from 0.11 to 0.28 in reading or English Language Arts (ELA) and from 0.18 to 0.41 in math (Kane & Staiger, 2012;Mihaly et al, 2013).…”

“…We consider these studies in two groups: those that analyze the relationships between observation-based measures and VAMs in a purely research context (e.g., Grossman, Loeb, Cohen, & Wyckoff, 2013;Hill, Kapitula, & Umland, 2011;Kane & Staiger, 2012;and Mihaly, McCaffrey, Staiger, & Lockwood, 2013) and those that assess these relationships as measured in the implementation of systems in practice (e.g., Kane, Taylor, Tyler, & Wooten, 2011;Sartain et al, 2011;TDOE, 2012). These studies vary in their implementation context (research versus practice), as well as in the observational measures employed (those that examine just the classroom-based aspects of observational measures and those that explore the whole measure) and in the methods used to estimate relationships between observational and test-based measures of teacher effectiveness.…”

“…As a result, despite federal and state incentives and policies that encourage or require the implementation of these new systems, there is as yet little empirical evidence regarding the usefulness of different measures to assess teachers' abilities to deliver high-quality instruction and improve student achievement in practice. Moreover, much of the extant research explores relationships between the two measures from a validity standpoint, attempting to assess if one measure is valid due to its correlation with other measures of effectiveness (e.g., Bell et al, 2012;Hill, Kapitula, & Umland, 2011;Holtzapple, 2003;Milanowski, 2004aMilanowski, , 2004b. There has been little research that specifically tackles the problem from the perspective of teachers and administrators implementing new evaluation systems, examining the different measures to see if the measures as they are given to teachers and principals in schools provide them with consistent signals about teacher effectiveness.…”

Mathematics Audit of the DoDEA Schools: 2014-2015

“…Studies have shown that higher MKT is positively correlated with a teacher's ability to select and enact mathematically rich tasks in elementary school classrooms (Charalambous, 2010), higher quality of instructional practices (Hill, Kapitula, and Umland, 2011), and student achievement in mathematics (Hill, Rowan, and Ball, 2005).…”

Ecologies of education quality

“…Gates, 2012;Wiggins, 2012;Winters, 2012). Skeptics of VAMs argue that we should be cautious in adopting the models, particularly if the aim is to use them to make comparisons between teachers or for individual personnel decisions because their reliability and validty are suspect (e.g., Au, 2007;Corcoran, 2010;Darling-Hammond, Amrein-Beardsely, Haertel, & Rothstein, 2012;Hill, Kapitula, & Umland, 2010). There is concern about how VAMs rely on standardized tests, which may not be up to the task of individual teacher evaluation (Amrein-Beardsley, 2009;Papay, 2010).…”