Observational Evaluation of Teachers: Measuring More Than We Bargained for?

Abstract

Our secondary analysis of Measures of Effective Teaching data contributes to growing evidence that observation ratings, used as part of comprehensive teacher evaluation systems across the nation, may measure factors outside of a teacher’s performance or control. Specifically, men and teachers in classrooms with high concentrations of Black, Hispanic, male, and low-performing students receive significantly lower observation ratings. By using various methodological approaches and a subsample of teachers randomly assigned to classrooms, we demonstrate that these differences are unlikely due to actual differences in teacher quality. These results suggest that policymakers consider the unintended consequences of using observational ratings to evaluate teachers and consider ways to adjust ratings to ensure they are fair.

Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations

“…To provide indirect evidence on how sensitive the MET protocols might have been to influences of social norms and process beyond the teachers' control, Table 4 reports regression models showing the association between compositional features of the classroom and protocol scores. As in Campbell and Ronfeldt (2018), we find that classroom achievement composition, racial composition, percentage free lunch, and even percentage male are associated with overall FFT scores (e.g. a coefficient of .091 for class mean achievement in our analyses is the same to two significant digits as estimated by Campbell and Ronfeldt).…”

“…For the fourth question, we provide indirect evidence on how sensitive the MET protocols might have been to influences of social norms and process beyond the teachers' control. We build closely off of Campbell and Ronfeldt's (2018) analysis of FFT to analyze the association between the teacher rating from each of the protocols, measured by averaging scores across subdomains, and classroom characteristics, including average prior test performance on ELA and math, percentage black, Hispanic, Asian, other non-white, male, special education, English language learners, free/reducedprice lunch, class size and teacher value-added. The extent to which classroom characteristics are individually or jointly significant predictors of FFT, CLASS, MQI or PLATO may be suggestive of influences beyond the teacher's control.…”

Can English Learner Teacher Effectiveness Be Observed? Validation of an EL‐Modified Framework for Teaching

“…We selected the FFT because it was used in our partner Local Education Agency district where we work on a longitudinal project that aims to build EL teacher expertise in school settings. We also noted recent research that evaluated the FFT and its ability to capture effective teacher practices (Campbell & Ronfeldt, 2018). Below we describe the process of adapting and implementing the FFT observation protocol for mainstream teachers of ELs.…”

“…Data from the tool reflects the two observable domains of instructional practice; the other two domains (Domains 1 and 4) were not observable during instruction. Other scholars have similarly examined teachers in single observations using rubrics limited to observable practices to high degrees of validity (Campbell & Ronfeldt, 2018;Pianta & Hamre, 2009;Taylor & Tyler, 2012).…”

Teacher Skill Development: Evidence from Performance Ratings by Principals

“…Recent research suggests that teachers’ classroom composition can influence the observation scores they receive (Campbell & Ronfeldt, ; Steinberg & Garrett, ; Whitehurst, Chingos, & Lindquist, ). Motivated by these findings, we also construct a residualized theta score by removing systematic variation explained by classroom‐ and school‐level average student demographic characteristics.…”