Learning As It Happens: A Decade of Analyzing and Shaping a Large-Scale Online Learning System

Abstract

With the advent of computers in education, and the ample availability of online learning and practice environments, enormous amounts of data on learning become available. The purpose of this paper is to present a decade of experience with analyzing and improving an online practice environment for math, which has thus far recorded over a billion responses. We present the methods we use to both steer and analyze this system in real-time, using scoring rules on accuracy and response times, a tailored rating system to provide both learners and items with current ability and difficulty ratings, and an adaptive engine that matches learners to items. Moreover, we explore the quality of fit by means of prediction accuracy and parallel item reliability. Limitations and pitfalls are discussed by diagnosing sources of misfit, like violations of unidimensionality and unforeseen dynamics. Finally, directions for development are discussed, including embedded learning analytics and a focus on online experimentation to evaluate both the system itself and the users' learning gains. Though many challenges remain open, we believe that large steps have been made in providing methods to efficiently manage and research educational big data from a massive online learning system. Notes for Practice• We analyzed an online adaptive practice environment for arithmetic, actively used by over 400,000 primary school children in the Netherlands.• Adaptive practice is achieved by continuously tracking both student abilities and item difficulties, and matching students to items.• A unidimensional adaptive algorithm, separately employed within each domain (e.g., multiplication), takes care of tracking abilities and difficulties.• We show that the obtained unidimensional ability and difficulty estimates are, to a large extent, reliable and accurate.• Moreover, we explore the many sources of misfit, or violations of the unidimensionality assumption, including differences in response processes (fast and slow responders) and response strategies (erroneous strategies that work for clusters of items).• To advance the field of learning analytics, we call for active analytics such as exemplified in this paper. Learning analytics must actively help direct a student towards his or her educational objective by means of embedded analytics that not only analyze the student, but also shape their learning path (such as the discussed adaptive algorithm) and includes experiments that ensure changes to the system have the desired effect.

Tracking Ability: Defining Trackers for Measuring Educational Progress

“…If ability θ is stable for some time, then changing, and then stable again, we can see how the KL divergence could decrease in times of stability and increase when ability changes. We believe this illustration shows that the convergence property is suitable for use in the practice of educational measurement, where students mostly respond to sets of items, even if they are assessed frequently (Brinkhuis et al 2018). The assumption here is that ability is stable during the relatively short time in which a student answers a set of items, and might change between the administrations of sets.…”

“…For now, we loosely define trackers as dynamic parameter estimates, adapting to possible changes in ability or item difficulty. Trackers can be especially useful in measurements that extend over a longer period of time at irregular time intervals, e.g., the continual measurement of abilities in computer adaptive practice (CAP) or computer adaptive learning (CAL) (Brinkhuis et al 2018;Klinkenberg et al 2011;Wauters et al 2010;Veldkamp et al 2011) or the monitoring of item difficulties in item banks (Brinkhuis et al 2015). Many actors can be involved in the possible changes of these parameters, including the pupils themselves, their teachers, their parents, educational reforms, etc.…”

Tracking with (Un)Certainty

“…To conclude, since CAL systems are becoming more widely adopted in education, the inferences made about individual children based on the reliable learning analytics provided by these systems should be a crucial part of designing the system (Brinkhuis et al 2018;Hofman et al 2018c). Based on these inferences tailored instructions could be provided, either automated within the system or by teachers in a classroom setting.…”

What Does Methodology Mean for Learning Analytics?

“…But (alternative explanation) what if the inferences are wrong, and the adaptive content selection is essentially arbitrary, or possibly even worse than chance? In this issue, Brinkhuis et al (2018) elaborate on several important model fit methodologies in the context of Math Garden, including the following: 1) visual inspection of residual differences between predicted and observed scores; 2) mixture modelling of residual differences to obtain corrected estimates of root mean square error; and 3) contingency table analysis, with constraints based on model specification.…”