Rater Performance Standards for Classroom Observation Instruments

Abstract

Raters must score accurately and consistently for classroom observation scores to be valid. This requires (a) a standard defining when scoring is accurate and consistent enough and (b) measuring and remediating rater performance against that standard. Current practice has focused on this second problem to the exclusion of the first. My goal here is to start a discussion about identifying a clear, explicit standard that ensures observation scores reflect a consistent view of teaching quality, rather than raters’ idiosyncratic perspectives. In doing so, I connect current certification test cut-scores, the current practice most analogous to a standard, to explicit rater standards, highlighting both the inadequacy of cut-scores and the low standards implicit to current practice.

Development of pre-service teachers’ teaching performance in physical education during a long-term internship

“…Classroom observation instruments have increasingly gained importance in recent years due to their potential to provide an evidence-based measure of teacher-stu-dent interactions and teaching performance (White, 2018). In contrast to low inferential rating systems, highly inferential rating systems require a high degree of inferences on the part of the observer (Lotz, Gabriel, & Lipowsky, 2013).…”

“…In contrast to low inferential rating systems, highly inferential rating systems require a high degree of inferences on the part of the observer (Lotz, Gabriel, & Lipowsky, 2013). Therefore, observers must be carefully trained in order to provide accurate and consistent ratings (White, 2018). One instrument that was developed and published in 2008 and has since been widely adopted for research and evaluation purposes in over 3000 early childhood and elementary classrooms is the Classroom Assessment Scoring System (CLASS; .…”

Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations

“…Overall, both the statistics in Table 3 and results published elsewhere show that in practice the reliability of classroom observations is quite variable, depends on adequate training and monitoring, and at the low end is problematic (Bell et al, 2014;Cash et al, 2012;Cohen & Gitomer et al, 2014;Goldhaber, 2016;Kane et al, 2012). White's (2018) analyses of the UTQ data suggest that current standards for rater accuracy and consistency may be too low. However, it would not be appropriate to label these protocols simply as "unreliable," as levels of agreement among expert raters are well above chance/independence (see quadratic weighted kappas) despite the inherent complexity of the phenomena being rated.…”

“…Some of these concerns have been discussed with regularity in the literature (#3; e.g. Bell et al, 2014;White, 2018), others became especially salient to us in our own analyses seeking to document interactions between instructional practices and classroom composition in the MET data (Aucejo et al, 2018), an investigation relevant to understanding the effects of between and within school sorting of students as well as teacher evaluation policies. In particular, concern #4 and #5 pose great difficulty in understanding how teachers might adapt instruction to match the needs of students.…”

Transcending Contemporary Obsessions

“…Campbell and Ronfelt (2018) more recently found that rates in measures of classroom observation protocols depend, in part, on factors beyond a teacher's performance or control, such as the teacher's gender and racial group, student population, and lower levels of student performance at the beginning of the year. White (2018) raised the issue of raters, concluding that there is a need for monitoring and re-training. For these reasons, we insist that the framework for lesson analysis should not be used as an evaluation tool.…”