ManyClasses 1: Assessing the generalizable effect of immediate versus delayed feedback across many college classes

Abstract

Psychology researchers have long attempted to identify educational practices that improve student learning. However, experimental research on these practices is often conducted in laboratory contexts or in a single class, threatening the external validity of the results. In this paper, we establish an experimental paradigm for evaluating the benefits of recommended practices across a variety of authentic educational contexts – a model we call ManyClasses. The core feature is that researchers examine the same research question and measure the same experimental effect across many classes spanning a range of topics, institutions, teacher implementations, and student populations. We report the first ManyClasses study, which examined how the timing of feedback on class assignments, either immediate or delayed by a few days, affected subsequent performance on class assessments. Across XX classes, [summarize effect of feedback timing, including key moderators]. More broadly, these findings provide evidence regarding the feasibility of conducting within-class randomized experiments across a range of naturally occurring learning environments.

Generalizability, transferability, and the practice-to-practice gap

“…In the case of applied research, we think researchers could be guided by the natural variation that occurs in actual practice. This was the strategy for our ManyClasses study (Fyfe et al, 2019). We examined how the timing of feedback on student work affected learning performance in 38 different college classes.…”

A Path to Greater Credibility: Large-Scale Collaborative Education Research

“…Multiteam collaborations may sound similar to traditional applications of meta-analysis, a tool to aggregate an existing body of work. Multiteam collaborations often make use of meta-analytic methods (e.g., weighted averaging of effect sizes, aggregating results of multiple studies), but because traditional meta-analyses include studies conducted across time, often by researchers applying different interventions with differing applications or protocols, and strongly affected by publication bias, the ability of meta-analysis to identify the true effect is limited (Fyfe, de Leeuw, Carvalho, Goldstone, & Motz, 2019). Traditional meta-analyses can rarely answer the question of how much of the variability in results or in replicability can be attributed to sample, researcher, statistical power, or bias (e.g., Lakens, Hilgard, & Staaks, 2016).…”

“…By having multiple research sites and collaborators, a single multiteam “study” can reveal what factors influence the effect better than a meta-analysis of independently conducted existing studies. Fyfe et al (2019) provide a relevant education example where the authors are using a multiteam approach to test the effect of timing of instructor feedback on subsequent class performance. Instead of using a single class with the same curriculum, the authors solicited a wide range of classes and disciplines.…”

“…For example, an individual researcher may only collect data in one state or one school district, thus limiting our understanding of whether we should expect results to generalize to other contexts. A consortium can facilitate assessment of generalizability and the existence of potential moderators on relevant results (see Fyfe et al, 2019, example above).…”

Actions for Increasing the Credibility of Educational Research

“…To that end, we created an osf webpage that documents the IRB protocol, the analysis plan, and instructor resources (https://osf.io/q84t7/). Further, the manuscript, which outlines the method and data collection details, was submitted as a registered report (Fyfe et al, 2019). Open communication is also a key priority.…”