Effect of Observation Mode on Measures of Secondary Mathematics Teaching

Abstract

Classroom observation of teachers is a significant part of educational measurement; measurements of teacher practice are being used in teacher evaluation systems across the country. This research investigated whether observations made live in the classroom and from video recording of the same lessons yielded similar inferences about teaching. Using scores on the Classroom Assessment Scoring SystemSecondary (CLASS-S) from 82 algebra classrooms, we explored the effect of observation mode on inferences about the level or ranking of teaching in a single lesson or in a classroom for a year. We estimated the correlation between scores from the two observation modes and tested for mode differences in the distribution of scores, the sources of variance in scores, and the reliability of scores using generalizability and decision studies for the latter comparisons. Inferences about teaching in a classroom for a year were relatively insensitive to observation mode. However, time trends in the raters' use of the score scale were significant for two CLASS-S domains, leading to mode differences in the reliability and inferences drawn from individual lessons. Implications for different modes of classroom observation with the CLASS-S are discussed.As the education reform movement increasingly focuses on teachers and teaching, educators, policymakers, and researchers need valid and reliable measures of teaching that can be used to evaluate individual teachers, provide guidance for improving teaching performance, and support research in ways that advance instruction and classroom dialog and practice. Nearly 20 years ago, Jaeger (1993) identified mode of observation as potentially contributing to the psychometric properties of measuring teaching, but little research on mode effects has occurred since. Renewed interest in measuring teaching and the large-scale use of observations for teacher evaluation systems has raised questions about the affordances of video capture, heightening the need for information on the comparability of scoring video and live observations. We present the first large-scale comparison of observation mode in the assessment of mathematics teaching.Observations of teaching are viewed as very useful data sources about teaching quality because they provide assessments that incorporate not only observation of the teacher's teaching but also the level of student engagement, the cognitive complexity of student-teacher interactions, and the subject matter focus and depth of instruction (Erickson, 2006;Jaeger, 1993). Video recording of classrooms is an alternative with practical advantages (Brunvard, 2010), especially with recent technological advances in the capture and transmission of digital audio and video.Extant research has shown very little difference in scores resulting from video and live observation. However, the studies were either not based on any rigorous evaluation (and were therefore inconclusive) or were conducted on data for nonclassroom contexts. Frederiksen, Sipusic, Gamoran, and Wolfe (1992) found ...

Liderazgo pedagógico en directores nóveles en Chile: Prácticas de observación de clases y retroalimentación a profesores

“…La observación en el aula y la retroalimentación se han convertido en prácticas de liderazgo pedagógico cada vez más comunes en varios sistemas educativos (Casabianca, McCaffrey, Gitomer, Bell, Hamre & Pianta, 2013;Ing & Montgomery, 2010;McMahon, Barrett & O'Neill, 2007;O'Leary, 2012;Peel, 2005;Taylor & Tyler, 2012). En un estudio realizado por la OCDE (2009) más del 70% de los profesores informaron que la observación en el aula era un aspecto importante en la evaluación de su desempeño.…”

“…Para avanzar hacia la objetivación en distintos países se han desarrollado diversos sistemas de observación de aula. Metodológicamente, el diseño de los sistemas de observación comprende cuatro principales desafíos: (a) la preparación del observador (Bell, Gitomer, McCaffrey, Hamre, Pianta & Qi, 2012;Ho & Kane, 2013); (b) las modalidades de observación (Casabianca et al, 2013;Mourshed, Chijoke, & Barber, 2010); (c) la frecuencia o número de observaciones durante un período de tiempo (Bloom, 2007;Leahy, 2012;Taylor & Tyler, 2012); y (d) el lugar desde donde proviene la observación (Haep, Bhnke & Steins, 2016). Se distingue entre orígenes externos e internos en términos de quién está conduciendo la observación.…”

Reactivity effects in video-based classroom research: an investigation using teacher and student questionnaires as well as teacher eye-tracking

“…Regarding the design, it also needs to be mentioned that our study focused on reactivity effects in video-based classroom research. The effects found might be different from the ones one would find if investigating reactivity regarding live observation (for differences in research findings between live and video observation, see Casabianca, McCaffrey, Gitomer, Bell, Hamre, & Pianta, 2013).…”

Advancing Human Assessment: A Synthesis Over Seven Decades

“…In addition, several contributions appeared by ETS authors in a three-volume handbook on IRT (Haberman 2016;von Davier and Rost 2016). Chapters by other researchers detail methods and statistical tools explored while those individuals were at ETS (e.g., Casabianca and Junker 2016;Moses 2016;Sinharay 2016).…”

“…Support was provided for testing programs, including an extensive series of job analyses for revising PRAXIS program assessments (Robustelli 2010) as well as work toward the development of new assessments (Phelps and Howell 2016;Sykes and Wilson 2015). The general topic of teacher evaluation remained a constant focus Goe 2013;Turkan and Buzick 2016), including continued investigation into implementing it through classroom observation (Casabianca et al 2013;Lockwood et al 2015;Mihaly and McCaffrey 2014) and value-added modeling (Buzick and Jones 2015;McCaffrey 2013;McCaffrey et al 2014). Researchers also explored the impact of teacher characteristics and teaching practices on student achievement (Liu et al 2010), the effects of professional development on teacher knowledge , and the connection between teacher evaluation and professional learning (Goe et al 2012).…”