Methodological Challenges in Conducting International Research on Teaching Quality Using Standardized Observations

Possible biases in observation systems when applied across contexts: conceptualizing, operationalizing, and sequencing instructional quality

“…However, as instruction tends to have similar patterns within national contexts (Hiebert et al, 2003), aspects of instruction on a micro level-for example, how teachers organize lesson activities-also have implications for how results in comparative cross-cultural studies are interpreted on a macro level. Therefore, context sensitivity in observation systems here refers to the extent that the operationalization of instructional features are broad enough to capture somewhat differently enacted practices of the same construct (Praetorius et al, 2019).…”

“…While interest in using standardized observation systems among classroom researchers is increasing, the majority of classroom observation studies use non-standardized and informal instruments (Bostic et al, 2019;Stuhlman et al, 2010), often in the forms of field notes and other inductive, in situ descriptions, arguing that the instrument should be sensitive to their specific research questions and research ambitions. This may be called a "bottom-up" approach to conceptualizing instructional quality, building the definition from the data (Praetorius et al, 2019). Such informal and non-standardized measurements of instruction may be useful for local purposes and for identifying features in classrooms that otherwise would be overlooked in a standardized manual.…”

“…Such informal and non-standardized measurements of instruction may be useful for local purposes and for identifying features in classrooms that otherwise would be overlooked in a standardized manual. However, they make it difficult to systematically capture and analyze patterns of instruction across multiple classrooms or aggregate consistent knowledge across studies, and may represent researchers' idiosyncratic understanding of quality instruction rather than an empirically driven systematic understanding (Praetorius et al, 2019).…”

“…Standardized observation systems have significant methodological benefits in general and in video studies in particular, as they facilitate a clear categorization and standardization of the massive amount of data that video studies tend to generate. Standardized observation systems are "top-down" approaches (Praetorius et al, 2019), consisting of pre-determined conceptualizations of instructional quality as lenses for understanding instruction in classrooms (Klette, 2015;Praetorieus & Charalombous, 2018). Broadly scoped, standardized observation instruments are useful because they can provide similar data across a broad range of contexts, and this supports the goal of integrating and accumulating knowledge (Klette & Blikstad-Balas, 2018).…”

“…A common international theory of instructional quality does not exist, while there are a number of different conceptualizations developed in certain cultural settings influenced by, for example, prioritized educational and teaching goals and specific subjects (Praetorius et al, 2019). Despite differences in views of learning underpinning different systems, scholars seem to agree upon some key features that are critical when measuring aspects of quality instruction (see, for example, Klette, 2015;Kunter et al, 2007;Nilsen & Gustafsson, 2016).…”