The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality

Abstract

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental study that manipulated the length of observation and order of presentation of 40-minute videotaped lessons from secondary grade classrooms. Results indicate that two 20-minute observation segments presented in random order produce the most desirable effect on score reliability and validity. This suggests that 20-minute occasions may be sufficient time for a rater to observe true characteristics of teaching quality assessed by the measure used in the study, and randomizing the order in which segments were rated may reduce construct irrelevant variance arising from carry over effects and rater drift.

Closing the Racial Discipline Gap in Classrooms by Changing Teacher Practice

“…It would be especially fruitful to ascertain whether students detect shifts in teacher practices across racial groups through student surveys and interviews. In addition, future research might include more than two observation points to strengthen claims that the sampled teacher practices adequately reflected somewhat stable or consistent behavior (Mashburn, Meyer, Allen, & Pianta, 2014). …”

Teacher interests, mastery goals, and self-efficacy as predictors of instructional practices and student motivation

“…Accordingly, teachers' reports of their motivation were more closely related to their self-rated practices than to student-rated practices, whereas students' motivation was more closely associated with their perceptions of instructional practices than with teacherrated practices. It follows that the perspective of an outside observer on teachers' instructional behavior (e.g., Lerkkanen et al, 2012;Mashburn, Meyer, Allen, & Pianta, 2014) might be helpful in order to get a less biased account of the relations among teacher motivation, instructional practices, and student motivation. In addition, the use of observational methods would potentially contribute to a better understanding of the nature of teacher and student reports of instruction and their discrepancy (Kunter & Baumert, 2006).…”

A Practical Solution to Optimizing the Reliability of Teaching Observation Measures Under Budget Constraints

“…Table 1 lists variance components for a two-facet and three-facet crossed design. We derived these variance components from a study of various observational designs (see Mashburn et al, 2013). This article focuses on the variance components for the instructional support scale of CLASS-S. 1 …”