What Are Error Rates for Classifying Teacher and School Performance Using Value-Added Models?

Abstract

This article addresses likely error rates for measuring teacher and school performance in the upper elementary grades using value-added models applied to student test score gain data. Using a realistic performance measurement system scheme based on hypothesis testing, the authors develop error rate formulas based on ordinary least squares and Empirical Bayes estimators. Empirical results suggest that value-added estimates are likely to be noisy using the amount of data that are typically used in practice. Type I and II error rates for comparing a teacher's performance to the average are likely to be about 25% with 3 years of data and 35% with 1 year of data. Corresponding error rates for overall false positive and negative errors are 10% and 20%, respectively. Lower error rates can be achieved if schools are the performance unit. The results suggest that policymakers must carefully consider likely system error rates when using value-added estimates to make high-stakes decisions regarding educators.Keywords: value-added models, performance measurement systems, student learning gains, false positive and negative error rates Student learning gains, as measured by students' scores on pretests and posttests, are increasingly being used to evaluate educator performance. Known as ''valueadded'' measures of performance, the average gains of students taught by a given teacher, instructional team, or school are often the most important outcomes for performance measurement systems that aim to identify instructional staff for special treatment, such as rewards and sanctions.Spurred by the expanding role of value-added measures in educational policy decisions, an emerging body of research has consistently found-using available data-that value-added estimates based on a few years of data can be imprecise. In this article, we add to this literature by systematically examining-from a design perspective-misclassification rates for commonly used performance measurement systems that rely on hypothesis testing.

Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement

“…I predict a teacher-level random effect as the value-added score. I utilize all years of data and all teachers in the sample districts and grades to increase the precision of my estimates (Goldhaber & Hansen, 2012;Koedel & Betts 2011;Schochet & Chiang, 2013). 6 In some instances, mean scores for both switchers and non-switchers on standardized variables fall below or above zero (e.g., Classroom Emotional Support).…”

Value-added modeling: A review

“…Of primary interest in many research studies is the real, persistent component of teacher quality, with the second and third factors contributing to the instability with which this primary parameter is estimated. 22 A number of studies have provided evidence on the stability of estimated teacher value-added over time and across schools and classrooms (e.g., see Aaronson, Barrow and Sander, 2007;Chetty, Friedman and Rockoff, 2014a;Glazerman et al, 2013;Goldhaber and Hansen, 2013;Jackson, 2014;Koedel, Leatherman and Parsons, 2012;McCaffrey et al, 2009;Schochet and Chiang, 2013). 21 This line of criticism is not specific to value-added.…”

Red Beads and Profound Knowledge: Deming and Quality of Education

“…Many empirical studies (see, for example, Haertel, 2013;Nye, Kanstantopoulos, & Hedges, 2004;and Schochet & Chiang, 2013) find that teachers account for less than 15% of the variability in test score gains. 4 Remember, by teachers I mean unexplained variability at the classroom level, so we cannot necessarily parse out the portion due to teachers and the portion due to other classroomlevel factors.…”