Replicating MOOC predictive models at scale

Gardner, Josh; Brooks, Christopher; Andrés, Juan; Baker, Ryan S.

doi:10.1145/3231644.3231656

Cited by 22 publications

(27 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We first recreated the state-of-the-art predictive model from Kovanović et al [15] using the original data and methodology, then applied insights from best practice when working with small data sets [16] to compare different algorithms for dealing with the unbalanced classes in the outcome variable. Building on these results, we explored the effect of splitting the data by course session instead of using a random split, in line with best-practice recommendations for replication studies in an educational context [8].…”

Section: Methodsmentioning

confidence: 99%

“…It is particularly important for automated classification techniques to be evaluated rigorously in order to understand how well they are likely to perform on new data. One notable recent development in this area is the MORF platform for replication of studies on MOOCs [8].…”

Section: Take Down Policymentioning

confidence: 99%

“…Using a random split to generate training data and test data from the same course(s) led to accuracy estimates that were significantly over-optimistic compared to testing on a later run of the course. Other work on replication [8] recommends using data from new sessions of the course for validation, rather than taking a random sample from the same course. This practice avoids the possibility of over-fitting to the training data.…”

Section: Best Practice In Training and Validating Modelsmentioning

confidence: 99%

“…Using stratified sampling to generate the training-test split allowed us to directly compare our results to prior work [15] when using a different approach to rebalancing the unbalanced classes. However, recent work on replicability of results on MOOCs [8] indicates that it is better to hold out the most recent offering of a course for testing. Even when the instructors are the same, a course changes every time it runs, with the difference between the first and second runs often being the greatest [7].…”

Section: Experiments 3: Comparing Data Splitsmentioning

confidence: 99%

See 3 more Smart Citations

Analysing discussion forum data

Farrow

Moore

Gašević

2019

Proceedings of the 9th International Conference on Learning Analytics &Amp; Knowledge

View full text Add to dashboard Cite

The widespread use of online discussion forums in educational settings provides a rich source of data for researchers interested in how collaboration and interaction can foster effective learning. Such online behaviour can be understood through the Community of Inquiry framework, and the cognitive presence construct in particular can be used to characterise the depth of a student's critical engagement with course material. Automated methods have been developed to support this task, but many studies used small data sets, and there have been few replication studies.In this work, we present findings related to the robustness and generalisability of automated classification methods for detecting cognitive presence in discussion forum transcripts. We closely examined one published state-of-the-art model, comparing different approaches to managing unbalanced classes in the data. By demonstrating how commonly-used data preprocessing practices can lead to over-optimistic results, we contribute to the development of the field so that the results of automated content analysis can be used with confidence.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Take Down Policymentioning

confidence: 99%

Section: Best Practice In Training and Validating Modelsmentioning

confidence: 99%

Section: Experiments 3: Comparing Data Splitsmentioning

confidence: 99%

See 2 more Smart Citations

Analysing discussion forum data

Farrow

Moore

Gašević

2019

Proceedings of the 9th International Conference on Learning Analytics &Amp; Knowledge

View full text Add to dashboard Cite

show abstract

“…In this line, there are few contributions. In one of them, Gardner et al [47] evaluated the results of a previous work of dropout prediction in other MOOCs and found that some research results (regarding algorithms and features used to predict) were the same while others indicated just the opposite. In order to get more insight about the factors influencing the predictions, this work will analyze the influence of different factors (mainly in RQ1) and several of them are replicated in two MOOCs as part of the methodology to delve into this issue.…”

Section: Related Workmentioning

confidence: 99%

Analysis of the Factors Influencing Learners’ Performance Prediction With Learning Analytics

et al. 2020

View full text Add to dashboard Cite

The advancement of learning analytics has enabled the development of predictive models to forecast learners' behaviors and outcomes (e.g., performance). However, many of these models are only applicable to specific learning environments and it is usually difficult to know which factors influence prediction results, including the predictor variables as well as the type of prediction outcome. Knowing these factors would be relevant to generalize to other contexts, compare approaches, improve the predictive models and enhance the possible interventions. In this direction, this work aims to analyze how several factors can make an influence on the prediction of students' performance. These factors include the effect of previous grades, forum variables, variables related to exercises, clickstream data, course duration, type of assignments, data collection procedure, question format in an exam, and the prediction outcome (considering intermediate assignment grades, including the final exam, and the final grade). Results show that variables related to exercises are the best predictors, unlike variables about forum, which are useless. Clickstream data can be acceptable predictors when exercises are not available, but they do not add prediction power if variables related to exercises are present. Predictive power was also better for concept-oriented assignments and best models usually contained only the last interactions. In addition, results showed that multiple-choice questions were easier to predict than coding questions, and the final exam grade (actual knowledge at a specific moment) was harder to predict than the final grade (average knowledge in the long term), based on different assignments during the course.

show abstract

A Brazilian Web Platform for Developing Massive Online Open Courses

Toledo

Seabra

2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Replicating MOOC predictive models at scale

Cited by 22 publications

References 27 publications

Analysing discussion forum data

Analysing discussion forum data

Analysis of the Factors Influencing Learners’ Performance Prediction With Learning Analytics

A Brazilian Web Platform for Developing Massive Online Open Courses

Contact Info

Product

Resources

About