A Randomized Experiment Testing the Efficacy of a Scheduling Nudge in a Massive Open Online Course (MOOC)

Abstract

An increasing number of students are taking classes offered online through open-access platforms; however, the vast majority of students who start these classes do not finish. The incongruence of student intentions and subsequent engagement suggests that self-control is a major contributor to this stark lack of persistence. This study presents the results of a large-scale field experiment (N = 18,043) that examines the effects of a self-directed scheduling nudge designed to promote student persistence in a massive open online course. We find that random assignment to treatment had no effects on near-term engagement and weakly significant negative effects on longer-term course engagement, persistence, and performance. Interestingly, these negative effects are highly concentrated in two groups of students: those who registered close to the first day of class and those with .edu e-mail addresses. We consider several explanations for these findings and conclude that theoretically motivated interventions may interact with the diverse motivations of individual students in possibly unintended ways.

Linking learner factors, teaching context, and engagement patterns withMOOClearning outcomes

“…Neglecting learner diversity in MOOCs may lead to ineffective or harmful interventions (Deng et al, 2020; Li & Baker, 2018). Baker et al (2016), for instance, found that an intervention designed to promote engagement had a negative effect on completion and performance, particularly for individuals who enrolled in the MOOC shortly before it began and those with ⋅edu email addresses.…”

“…Ferguson et al (2015) found that learners classified as “keen completers” in an eight‐week MOOC fully engaged in most weeks, posted threads in discussion boards and completed assessments on time, while participants labelled as “keen completers” in a three‐week MOOC did not exhibit all these traits in any week. Baker et al (2016) reported that a scheduling nudge designed to promote engagement had weakly significant, moderately negative effects on the number of video lectures watched by MOOC learners. Rogers and Feller (2016) revealed that learners who graded peers' excellent essays in a peer assessment were more likely to quit than those who assessed peers' less exceptional essays.…”

“…Correlational studies have also shown how configurations of MOOC teaching environments can affect engagement (e.g. Baker, Evans, & Dee, 2016) and how engagement can affect learning outcomes (e.g. Phan, McNeil, & Robin, 2016).…”

The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes

“…Studies examining the effects of externally and self-imposed interim deadlines on course grades have found positive (e.g., Ariely and Wertenbroch, 2002), negative (e.g., Burger, Charness, and Lynham, 2011), and null effects (e.g., Levy and Ramim, 2013). Studies examining the effects of encouraging students to plan when they will do work have also found a mix of positive (Baker, Evans, Li, and Cung, 2019), negative (Baker, Evans, and Dee, 2016), and null (Sitzmann and Johnson, 2012) effects on course and assignment grades.…”

When Studying and Nudging Don’t Go as Planned: Unsuccessful Attempts to Help Traditional and Online College Students

“…7 Our paper also relates to the recent literature on pre-commitment devices in higher education. Himmler et al (2017) find that asking students to pre-commit to taking exams at a certain time improved their overall performance in a graduate business school, while Baker et al (2016) find that prompting students via email to schedule times to watch lecture videos in a Massive Open Online Course had no impact on performance. In a similar setting, Patterson (2015) finds that enabling students to pre-commit to limits on distracting internet time increased completion rates and improved performance but reminding students about their coursework and allowing students to block distractions while working had no significant impact on course outcomes.…”