Using Adaptive Experiments to Rapidly Help Students

Abstract

Adaptive experiments can increase the chance that current students obtain better outcomes from a field experiment of an instructional intervention. In such experiments, the probability of assigning students to conditions changes while more data is being collected, so students can be assigned to interventions that are likely to perform better. Digital educational environments lower the barrier to conducting such adaptive experiments, but they are rarely applied in education. One reason might be that researchers have access to few real-world case studies that illustrate the advantages and disadvantages of these experiments in a specific context. We evaluate the effect of homework email reminders in students by conducting an adaptive experiment using the Thompson Sampling algorithm and compare it to a traditional uniform random experiment. We present this as a case study on how to conduct such experiments, and we raise a range of open questions about the conditions under which adaptive randomized experiments may be more or less useful.

Amortised Design Optimization for Item Response Theory

“…A considerable amount of recent work has focused on inferring IRT parameters using [2], who propose a personalized exercise recommendation system for an online self-directed learning service, and Dynamic Knowledge Embedding and Tracing by Xu et al [142]. Within the context of experimental design in education, Zavaleta-Bernuy et al [144] suggest an adaptive method for designing experiments by using Thompson sampling. In Publication IV, the proposed solution formulates the joint experimental design and parameter estimation task as a Partially Observable Markov Decision Process (POMDP).…”