Evaluating Special Educators’ Classroom Performance: Does Rater “Type” Matter?

Abstract

Classroom observations remain the predominant data source used in teacher evaluations, but little is known about how rater characteristics may affect teachers' scores. For special educators, whose instructional practice requires specialized knowledge and skills, school administrators (i.e., the raters) without experience in special education teaching may not be able to provide reliable and unbiased scores. This study included three school administrators who viewed and scored the classroom teaching of 19 special educators in California and Idaho; individuals with special education teaching experience also scored a subset of the instructional lessons. Generalizability study analyses revealed that the school administrators were not consistently reliable scorers of the special education teachers' instruction. The school administrators were also more lenient on average in their ratings than peer teacher raters. Findings from this preliminary investigation suggest that rater background may matter, and future studies should explore how specific rater characteristics such as background and experience affect special educators' performance ratings.

Policies That Define Instruction: A Systematic Review of States’ and Districts’ Recommendations for Evaluating Special Educators

“…Preobservation conferences are promoted or required in many evaluation systems (Steinberg & Donaldson, 2016), but there is evidence that they may not take place due to time constraints (Kraft & Gilmour, 2016). Additionally, even if school leaders use preobservation conferences to customize evaluation systems, evaluators without special education experience struggle with evaluating SETs, primarily SETs’ use of instructional practices aligned with effective instruction for SWDs (Lawson & Cruz, 2018a, 2018b). Flexibility resulting in effective and appropriate modifications requires a knowledgeable evaluator, a knowledgeable SET, and a cooperative relationship between the evaluator and the SET.…”

Observation Studies in Special Education: A Synthesis of Validity Evidence for Observation Systems

“…Chow and Kasari (1999) described conducting preliminary analysis for differences in observation results related to teacher gender or the school at which the observation occurred. Examining bias related to experience in special education, the purpose of Lawson and Cruz's (2018) study was to examine differences in observation scores between observers with and without special education experience. E. S. Johnson and Semmelroth (2015) examined the factors that contributed to variance in observation scores; one of those factors was the rater.…”

“…H. L. Swanson (1984) ensured no observer was at the same school more than once and assigned two observers to each observation. Lawson and Cruz (2018) assigned two observers to each video and randomized which video each pair of observers coded. In most studies involving RESET (e.g., Crawford et al, 2019;E.…”

“…In other words, this may be a result of insufficient researcher knowledge. We recommend scholars look to some of the stronger studies in our corpus as exemplars of how to design an observation system to support both accuracy and consistency (e.g., Lawson & Cruz, 2018;Swank et al, 1989;Wexler et al, 2018). For example, Wexler et al (2018) depict how they trained and supported observers, providing evidence that all scores were attuned to the same standard of measurement.…”

Validasi Modul “GuLaLi” untuk Meningkatkan Keterampilan Guru Mengajar Siswa Tunarungu Perlindungan Diri dari Kekerasan Seksual

“…Bias dan kesalahan pengamatan dipengaruhi oleh tingkat pengalaman rater, pengetahuan dan keterampilan (Davis, 2016). Terkadang rater memberi penilaian tidak konsisten terhadap instruksi yang diberikan (Lawson and Cruz, 2017). Oleh karena itu, tingkat kesepakatan antar rater perlu dikaji untuk memastikan bahwa hasil observasi tersebut objektif dan dapat dipercaya.…”