Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations

Kelly, Sean; Bringe, Robert; Aucejo, Esteban; Fruehwirth, Jane Cooley

doi:10.14507/epaa.28.5012

Cited by 27 publications

(26 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This effectively limits the range of teacher intentions and learning goals that will be encountered when using IQA, which in principle allows the IQA to be more carefully tailored to those teacher intentions. If one takes a strong view about the importance of teacher intentions, measuring teaching quality accurately may require using specialized systems that narrowly capture only key aspects of teaching quality (Kelly et al, 2020).…”

Section: Assumption 1: Teaching Is An Intentional Activitymentioning

confidence: 99%

Bringing the Theory and Measurement of Teaching into Alignment

White¹,

Luoto²,

Klette³

et al. 2021

Preprint

View full text Add to dashboard Cite

There is a growing use of standardized observation systems to directly measure teaching quality in classrooms. These systems are based on rich, theoretical understandings of teaching quality that lead to carefully operationalized rubrics that decompose teaching quality into a number of distinct dimensions. In this paper, we argue that many of the ways observation systems are applied and analyzed do not fully conform to the theoretical understandings of teaching quality upon which the observation system is based. We further argue for the need to align theoretical understandings of teaching quality with analytical approaches to measure teaching quality. We discuss four key assumptions that undergird many views of teaching quality and highlight how common analytical approaches violate these assumptions, proposing alternative analytical approaches that would better conform to the theoretical understandings of teaching quality. We end with a discussion of the importance of carefully aligning theoretical understandings with measurement approaches.

show abstract

Section: Assumption 1: Teaching Is An Intentional Activitymentioning

confidence: 99%

Bringing the Theory and Measurement of Teaching into Alignment

White¹,

Luoto²,

Klette³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To that end, we believe that equity analytics and the development of tools to capture the distribution of opportunities to learn such as EQUIP (https://www.equip.ninja) offer some promise for answering the questions we mentioned above using equity ratios and have been used to examine how teachers might use such data to address implicit bias (Reinholz et al, 2020). Furthermore, such measures may be positioned well to measure teacher adaptation and change (Kelly et al, 2020). However, fine-grained observation instruments, including dyadic observation instruments, are often expensive and difficult to collect and implement (Kelly et al, 2020).…”

Section: Technology Toolsmentioning

confidence: 99%

“…Furthermore, such measures may be positioned well to measure teacher adaptation and change (Kelly et al, 2020). However, fine-grained observation instruments, including dyadic observation instruments, are often expensive and difficult to collect and implement (Kelly et al, 2020). For those reasons, we imagine that numerous tools and applications can be developed to measure and provide feedback on who participates in classrooms and the nature of that participation.…”

Section: Technology Toolsmentioning

confidence: 99%

Using dyadic observation to explore equitable learning opportunities in classroom instruction

Lavigne

Good

2021

EPAA

View full text Add to dashboard Cite

Because of poverty, many children do not receive adequate prenatal care, nutrition, or early childhood education. These inequities combine to ensure that many students enter school with considerably less academic content knowledge and skills for learning than their peers. Teachers and schools did not create these gaps, but they must address them. The impact of schools in reducing gaps has been explored for decades only to yield inconsistent findings. One possible reason for these contradictory results is because these studies ignore classroom process. We argue for the inclusion of process in research on opportunity and achievement gaps to better articulate if schools provide inequitable learning opportunities. Further, we argue for dyadic (teacher to individual student) measurement of classroom process because commonly-used observation instruments only measure teachers’ interactions with the whole class. These instruments obscure differential teacher treatment that may exist in some classrooms. To improve policy and practice, we call for supplementing extant measures of teachers’ whole-class interactions (process) and student outcome (product) measures with those that measure dyadic interactions to learn how opportunities to learn in classrooms and schools are distributed among students to reduce, sustain, or enhance learning gaps.

show abstract

“…Rating quality was further enhanced with calibration videos at the beginning of each rating session, by interspersing "validity videos" into each rater's workload, and by consultation with scoring leaders who "back-scored" a sub-sample of videos to identify raters who needed additional training. Despite generally adequate training and quality control for large-scale research purposes, several measurement limitations have been identified in the MET study, including imprecise discrimination in lesson quality, lack of independence in sub-domains, and sensitivity to rater training (see Kelly et al, 2020, for full discussion of limitations). Yet, at the present time we would still consider the six measures we use to be state-of-the-art measures of instruction.…”

Section: Observation and Instructional Measurement Processmentioning

confidence: 99%

“…The protocols used in MET are "global" protocols (Kelly et al, 2020), applying overall scores to multiple domains of practice from viewing segments of instruction. For example, CLASS is divided into three broad domains, emotional support, classroom organization, and instructional support, with those in turn made up of more specific subdomains (e.g., quality of feedback).…”

Section: Nature Of the Observational Measuresmentioning

confidence: 99%

School-to-School Differences in Instructional Practice: New Descriptive Evidence on Opportunity to Learn

Kelly

Mozenter²,

Aucejo

et al. 2020

Teachers College Record

Self Cite

View full text Add to dashboard Cite

Background/Context There is continuing debate among social scientists and educators about the role of school-to-school differences in generating educational inequality. Are some students high achieving because they attend School A, while others struggle because they attend School B, as critical discourse on schools argues? Alternatively, is educational inequality driven largely by social forces outside of the school, in the home and neighborhood environment, or by educational processes that are largely common across schools as much social science research argues? Analyses of school achievement, and in particular test score gains from year-to-year, suggest very small between-school differences. Yet, analyses of test score data alone may fail to reveal important school-to-school differences that affect the quality of the classroom experience and a variety of educational outcomes. Purpose/Objective We provide evidence on the following research questions. What is the magnitude of school-to-school variation in instructional practice, as captured by multiple measures? Are some domains of instruction (e.g., behavioral management) more variable between schools than others? To what extent are school-to-school differences in instruction associated with compositional characteristics of students and teachers? Research Design This study relies on the Measures of Effective Teaching Study data, which offer an unprecedented set of observations of teachers’ instruction scored on state-of-the-art observational protocols. To examine the extent of school-to-school variation in instructional practice in elementary and middle schools, we conducted a decomposition of variance analysis using summary scores on multiple measures. We further examine behavioral climate as revealed during instruction separately from overall instructional practice. Next, we examine differences in instruction associated with compositional characteristics of students using multilevel models. Finally, we use an innovative two-stage statistical adjustment strategy to more narrowly identify the possible association between composition and teaching practice due to school-to-school teacher sorting. Findings/Results The basic descriptive results from this study suggest a middle view of school-to-school differences in instruction. We find that substantial school-level variation in instruction exists, with 30% or more of the total variance in instruction lying between schools in these data. Behavioral climate during instruction appears to be particularly salient, and especially in elementary schools. Much of the between school variance we identify, in some cases 40% or more, is readily explained by simple measures of socio-demographic composition, including in particular the racial make-up of schools in the MET districts. Finally, some evidence from a statistical adjustment method suggests that teacher sorting, rather than measurement bias and teacher adaptation, is principally responsible for school-to-school differences in instruction. Conclusions/Recommendations More than an academic debate, basic differences between schools in the quality of the learning environment, along with parental understandings and beliefs about school effects, are potentially important drivers of school and neighborhood sorting and segregation, and even public investment in schooling. Additionally, this question carries continued policy relevance as states adopt and revise teacher and school accountability frameworks that implicitly attribute school-to-school differences to organizational functioning, and seek to carry out instructional improvement efforts in targeted schools. The basic descriptive results from this study suggest school-level differences are not as great as suggested by critical theory and the public discourse, but neither are they as inconsequential as one might infer from some social science research or the literature on value-added differences between schools.

show abstract

Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations

Cited by 27 publications

References 56 publications

Bringing the Theory and Measurement of Teaching into Alignment

Bringing the Theory and Measurement of Teaching into Alignment

Using dyadic observation to explore equitable learning opportunities in classroom instruction

School-to-School Differences in Instructional Practice: New Descriptive Evidence on Opportunity to Learn

Contact Info

Product

Resources

About