The quality and qualities of classroom observation systems

Abstract

A teacher survey can provide insight into teachers' own perceptions of their teaching quality and their intentions, thought processes, knowledge, and beliefs. Teachers also have, in contrast to external raters, full knowledge of the classroom context, for example regarding the background of the performance of specific students (Goe et al., 2008). Although teacher surveys can stimulate teacher reflection, underperforming teachers might not have the metacognitive competence to recognize their professional skills and high performing teachers might underestimate these skills (Kruger & Dunning, 1999). Another way of measuring teaching practices is by means of instructional collections and artifacts. An instructional collection, also called a portfolio, is a collection of materials that is compiled by teachers to provide evidence of their fulfillment of predetermined standards (Goe et al., 2008). Examples of such evidence are lesson plans, assignments, reflective writings, and samples of student work (Gitomer & Bell, 2010). An artifact protocol is a much narrower type of instructional collection, and can for example be focused on the quality of the student assignments that teachers provide. Building an instructional collection can stimulate teacher reflection and can help them improve. It provides insight into the learning opportunities for students on a day-today basis. However, it is a time-consuming enterprise for both teachers and assessors, and one might question to what extent teachers' exemplary work reflects their everyday classroom activities. Another teaching quality measure, in which not the teaching process but the outcomes of teaching are evaluated, is a measure of a teacher's added value (Goe et al., 2008). Since schools and teachers teach different student populations, it would be unfair to compare solely the performance of the students across teachers. In value-added models, students' prior educational attainment, their background characteristics, and the school composition are often taken into account to make comparisons of teachers' output more fair (Timmermans, Bosker, Doolaard, & de Wolf, 2012). Such measures enable the evaluation of teachers' contribution to student learning in a costeffective and non-intrusive way, since most of the required data (test scores) have already been collected for other purposes (Goe et al., 2008). However, the use of value-added models in the context of education is a controversial issue. A review 1.5 A framework in support of the development, selection, and use of COSs Recent publications show that generating valid and reliable scores by means of a COS is not self-evident (e.g.,

Unravelling the challenges of the data-based approach to teaching improvement

“…As teachers vary in quality (Haertel, 2013;Hanushek & Rivkin, 2012;Nye et al, 2004), as expressed in how much their students learn, teachers' teaching skills must also differ. Referring to the characteristics of effective teaching that were mentioned in the previous paragraphs, teachers differ for example in their classroom management skills, in how well they explain subject matter to students, and in the skill to adapt their teaching in line with student needs (e.g., Dobbelaer, 2019).…”

The Quality of Student Perception Questionnaires: A Systematic Review

“…The contacted field experts were all researchers conducting research into teaching evaluation, or had developed or used an SPQ in their research. A total of 92 researchers from 13 different countries were contacted for the purpose of both a systematic review of classroom observation systems (Dobbelaer, 2019) and for the current study. They were asked to name SPQ that met the inclusion criteria.…”

“…In line with Dobbelaer (2019), who reviewed classroom observation systems, the creation of an international database with all SPQs, that have been developed, their materials and the research into them, could make SPQ developers more aware of the value of their SPQ for others. If SPQ materials are accessible to others, SPQs can also be researched and developed further (even if the instrument developers themselves did not have the resources to do so), which could also help to reduce the need to constantly develop new SPQs.…”

“…Therefore, ideally, an SPQ also includes a user manual, scoring rules, and sampling specifications (for example, specifications regarding the subject of the lesson). Although the use of SPQs is not new, several studies (Bijlsma et al, under review;Lüdtke et al, 2006;van der Scheer et al, 2018;Wallace et al, 2016) have recently generated renewed interest in reliability and validity issues surrounding SPQs and other teacher evaluation approaches, such as classroom observation systems (Bell et al, 2018;Dobbelaer, 2019).…”

The Conceptualization Implies the Statistical Model: Implications for Measuring Domains of Teaching Quality

“…The direct measurement of teaching quality through observation systems (Bell et al, 2019) has become an important tool in research (e.g., Dobbelaer, 2019) and practice (e.g., Jones & Bergin, 2019). Observation systems provide a specific conceptualization of teaching quality, operationalized across a set of measured dimensions that are typically grouped under a small number of domains that are thought to be the active ingredients for supporting student learning.…”