MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data

Abstract

Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an opensource platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains.

Algorithmic Bias in Education

“…Of course, as discussed in section 4.1, our recommendation to collect richer identity data does have the drawback of increasing risk around privacy and increased regulatory challenges. Approaches such as data obfuscation (Bakken et al, 2004), providing researchers the ability to use but not view variables (Gardner et al, 2018), legal agreements around data re-identification (ASSISTments Project, 2014), can mitigate these risks to a degree. Encouraging regulators and institutional review boards (or other privacy officers) to balance the risks of privacy violations with the risks of algorithmic bias will also be highly important.…”

Algorithmic Bias in Education

“…Benitez & Malin, 2010); a classroom may only have one female indigenous student, and therefore reporting this information creates a serious privacy risk. There are methods that can be used to reduce this risk, such as data obfuscation (Bakken et al, 2004), providing researchers the ability to use but not view variables (Gardner et al, 2018), legal agreements around data reidentification (ASSISTments Project, 2014), but the risk is hard to entirely eliminate in a small data set.…”

Multiplatform MOOC Analytics: Comparing Global and Regional Patterns in edX and Edraak

“…Much of MOOC research over the past five years has been conducted in studies of single higher education institutions, and even the largest studies have aggregated data from within a single MOOC provider. One of the most promising initiatives in this area is MORF framework [10], a platform that enables institutions to securely deposit their MOOC data and allows researchers to execute Docker containers for data analysis while maintaining full privacy of the data. In this study, we propose a methodology that we denote as multiplatform MOOC analytics, which leverages commonalities across MOOC learning and content management systems to allow research teams to create common data formats, agree upon analytic methods, and then generate aggregate data, produced through identical processes, that can allow for "apples-to-apples" comparisons between different MOOC platforms.…”