State and Local Efforts to Investigate the Validity and Reliability of Scores from Teacher Evaluation Systems

Abstract

Context In the past two years, states have implemented sweeping reforms to their teacher evaluation systems in response to Race to the Top legislation and, more recently, NCLB waivers. With these new systems, policymakers hope to make teacher evaluation both more rigorous and more grounded in specific job performance domains such as teaching quality and contributions to student outcomes. Attaching high stakes to teacher scores has prompted an increased focus on the reliability and validity of these scores. Teachers unions have expressed strong concerns about the reliability and validity of using student achievement data to evaluate teachers and the potential for subjective ratings by classroom observers to be biased. The legislation enacted by many states also requires scores derived from teacher observations and the overall systems of teacher evaluation to be valid and reliable. Focus of the Study In this paper, we explore how state education officials and their district and local partners plan to implement and evaluate their teacher evaluation systems, focusing in particular on states’ efforts to investigate the reliability and validity of scores emerging from the observational component of these systems. Research Design Through document analysis and interviews with state education officials, we explore several issues that arise in observational systems, including the overall generalizability of teacher scores; the training, certification, and reliability of observers; and specifications regarding the sampling and number of lessons observed per teacher. Findings Respondents’ reports suggest that states are attending to the reliability and validity of scores, but inconsistently; in only a few states does there appear to be a coherent strategy regarding reliability and validity in place. Conclusions There remain a variety of system design and implementation decisions that states can optimize to increase the reliability and validity of their teacher evaluation scores. While a state may engage in auditing scores, for instance, it may miss the gains to reliability and validity that would accrue from periodic rater retraining and recertification, a stiff program of rater monitoring, and the use of multiple raters per teacher. Most troublesome are decisions about which and how many lessons to sample, which are either mandated legislatively, result from practical concerns or negotiations between stakeholders, or, at best case, rest on broad research not directly related to the state context. This suggests that states should more actively investigate the number of lessons and lesson sampling designs required to yield high-quality scores.

Can Teacher Evaluation Systems Produce High-Quality Feedback? An Administrator Training Field Experiment

“…We grouped eligible schools into six blocks based on school size (above or below the median of 390 students) and type (elementary, middle, or high) and then randomized within the six school size-type blocks. We chose these blocks based on prior research that suggests there are systematic differences in school climate and teacher working conditions by school size and type (Herlihy et al, 2014; Lee & Loeb, 2000). We also hypothesized that administrators at larger schools may face more capacity constraints because they are assigned to evaluate more teachers.…”

“…Relational trust between administrators and teachers plays a key role in evaluation and improvement (Bryk et al, 2010; Bryk & Schneider, 2002; Tuytens & Devos, 2010, 2014). Teachers are more willing to recognize their weaknesses and try new instructional approaches when school leaders establish a culture of continuous improvement (Herlihy et al, 2014). Teachers are also more likely to respond to feedback in productive ways when they perceive the feedback as valid and worth acting on; otherwise, they may ignore it or implement it in superficial ways (Lane, 2020; Spillane et al, 2002).…”

“…Administrators themselves have expressed strong interest in receiving professional development on coaching teachers (Kraft & Gilmour, 2016; Sporte et al, 2013). However, what little training administrators have received as part of new evaluation systems has largely been focused on improving their reliability as classroom observers for accountability purposes (Herlihy et al, 2014; Bell et al, 2016). We still know very little about how to improve the quality of feedback administrators provide.…”

The Change We Cannot See: Instructional Quality and Classroom Observation in the Era of Common Core

“…Determining the unit of analysis is an integral part of every measurement decision because it indicates the level at which conclusions will be drawn (Cronbach et al, 1972). Protocols rely on different sampling parameters to generalize about a teacher’s “practice” (Bell et al, 2012; Herlihy et al, 2014; Hill et al, 2012). Despite considerable work over the last decade, questions remain about the ability to make accurate inferences about instructional quality from classroom observation (Cohen & Goldhaber, 2016).…”

Implementation of a new teacher evaluation program : principal and teacher perceptions

“…Knowledge or lack of knowledge also influences the implementation process. A study by Herlihy et. al.…”