Studying MOOC completion at scale using the MOOC replication framework

Andrés, Juan; Baker, Ryan S.; Gašević, Dragan; Siemens, George; Crossley, Scott A.; Joksimovíc, Srécko

doi:10.1145/3170358.3170369

Cited by 30 publications

(31 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[64]) or on replicating findings from one course in another (e.g. [3]), not on developing models that can be used across classes or class offerings. This is crucial because in order for a model to be useful it must be the case that we can train it before it is actually necessary.…”

Section: Introductionmentioning

confidence: 99%

How Widely Can Prediction Models Be Generalized? Performance Prediction in Blended Courses

Gitinabard

Heckman

et al. 2019

IEEE Trans. Learning Technol.

View full text Add to dashboard Cite

Blended courses that mix in-person instruction with online platforms are increasingly popular in secondary education. These tools record a rich amount of data on students' study habits and social interactions. Prior research has shown that these metrics are correlated with students' performance in face to face classes. However, predictive models for blended courses are still limited and have not yet succeeded at early prediction or cross-class predictions even for repeated offerings of the same course. In this work, we use data from two offerings of two different undergraduate courses to train and evaluate predictive models on student performance based upon persistent student characteristics including study habits and social interactions. We analyze the performance of these models on the same offering, on different offerings of the same course, and across courses to see how well they generalize. We also evaluate the models on different segments of the courses to determine how early reliable predictions can be made. This work tells us in part how much data is required to make robust predictions and how cross-class data may be used, or not, to boost model performance. The results of this study will help us better understand how similar the study habits, social activities, and the teamwork styles are across semesters for students in each performance category. These trained models also provide an avenue to improve our existing support platforms to better support struggling students early in the semester with the goal of providing timely intervention.

show abstract

Section: Introductionmentioning

confidence: 99%

How Widely Can Prediction Models Be Generalized? Performance Prediction in Blended Courses

Gitinabard

Heckman

et al. 2019

IEEE Trans. Learning Technol.

View full text Add to dashboard Cite

show abstract

“…The limited MOOC replication research to date has shown that many published findings in the field may not replicate. For example, in a large-scale replication, [14] found that only 12 of 15 previously-published MOOC findings replicated significantly across the data sets, and that two findings replicated significantly in the opposite direction; similar results in a machine learning replication in the context of algorithm and feature selection were shown in [15].…”

Section: B Lack Of Replication Studies In Learning Sciencesmentioning

confidence: 68%

“…The open availability of executable machine learning experiments affords detailed meta-analyses by providing complete results of all modeling stages for meta-analysis. MORF has already been used for metaanalysis [14].…”

Section: B Beyond Verification: the Benefits Of Replicationmentioning

confidence: 99%

MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data

Gardner

Brooks

Andrés

et al. 2018

2018 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an opensource platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains.

show abstract

“…Other work has included information outside the clickstream, such as forum post data analyzed with natural language processing [10,17]. Frameworks for evaluating various models and their feature sets have also been introduced [15,22]. While most of these studies focused on only a small set of courses, [11,22] used a dataset of 20 or more courses to train and predict.…”

Section: Related Workmentioning

confidence: 99%

“…Frameworks for evaluating various models and their feature sets have also been introduced [15,22]. While most of these studies focused on only a small set of courses, [11,22] used a dataset of 20 or more courses to train and predict.…”

Section: Related Workmentioning

confidence: 99%