The Unintended Effects of Policy-Assigned Teacher Observations: Examining the Validity of Observation Scores

Abstract

Several state policies link high-stakes consequences to teacher evaluations, which tend to be heavily weighted by observation scores. However, research has only recently investigated the validity of these scores in field settings. This study examines the sensitivity of teacher observation scores to the number of observations assigned by state policy and the assignment of prior-year composite measures produced by the evaluation system. Regression discontinuity and local regression designs exploit discontinuities in both assignment processes. The evidence suggests that assignment to a lower prior-year composite score does not bias observation scores, but the assignment to more policy-assigned observations introduces substantial negative bias. The degree of negative bias is most pronounced among early-career teachers, as suggested by theory. Implications are discussed.

Critical Feedback Characteristics, Teacher Human Capital, and Early-Career Teacher Performance: A Mixed-Methods Analysis

“…Furthermore, the negative main and moderated associations with observation scores may reflect bias. Research suggests that an observer's prior knowledge about a teacher's performance can influence observation scores (Hunter, 2020). Observers might issue critical feedback to teachers they expect will have difficulty improving; these expectations may downwardly bias subsequently assigned observation scores independently of teachers' performance.…”

“…Another widely used approach to measuring teacher quality is classroom observation. Although classroom observations have been used to assess teachers for far longer than VAM (Brophy & Good, 1986), researchers have only recently begun to evaluate the properties of observation scores (Campbell & Ronfeldt, 2018; Hunter, 2020; Steinberg & Garrett, 2016). The depth of research examining the connections between teacher observation scores and students’ outcomes is much more limited.…”

“…Per year, TEAM policy assigns a minimum of four observations to teachers receiving Tennessee’s lowest teacher effectiveness score, four or two observations to teachers in the middle categories of effectiveness depending on their certification, and one observation to teachers receiving the highest effectiveness score (Teacher and Principal Evaluation Policy, 2013). Although state policy expects the typical observation to last approximately 15 minutes (Teacher and Principal Evaluation Policy, 2013), teachers report that observations tend to last about 30 minutes (Hunter, 2020). School administrators decide which observers will conduct observations of which teachers.…”

Explaining variation in the implementation of observational processes by school leaders: evidence from a Tennessee-based researcher–practitioner partnership

“…The administrative data also included the principal and assistant principal's total years of experience and an administrator version of prior-year LOE based on school-level student outcomes and observation scores based on the school administrator's performance portfolio (for details, see Grissom et al. , 2018, and Hunter, 2020). Whether the observer was a principal or assistant principal was unknown.…”

“…The least effective teachers are assigned a minimum of four observations and the most effective a minimum of one. Moderately effective teachers are Variation in observational processes assigned four or two observations depending on their years of experience (for details see Hunter, 2020). Teachers report that observations last approximately 30 min (Hunter, 2020).…”