How Widely Can Prediction Models Be Generalized? Performance Prediction in Blended Courses

Abstract

Blended courses that mix in-person instruction with online platforms are increasingly popular in secondary education. These tools record a rich amount of data on students' study habits and social interactions. Prior research has shown that these metrics are correlated with students' performance in face to face classes. However, predictive models for blended courses are still limited and have not yet succeeded at early prediction or cross-class predictions even for repeated offerings of the same course. In this work, we use data from two offerings of two different undergraduate courses to train and evaluate predictive models on student performance based upon persistent student characteristics including study habits and social interactions. We analyze the performance of these models on the same offering, on different offerings of the same course, and across courses to see how well they generalize. We also evaluate the models on different segments of the courses to determine how early reliable predictions can be made. This work tells us in part how much data is required to make robust predictions and how cross-class data may be used, or not, to boost model performance. The results of this study will help us better understand how similar the study habits, social activities, and the teamwork styles are across semesters for students in each performance category. These trained models also provide an avenue to improve our existing support platforms to better support struggling students early in the semester with the goal of providing timely intervention.

Predicting Student Academic Performance by Means of Associative Classification

“…The results achieved on the Moodle LMS data and presented in [17,18] show that student-student and studentteacher interactions are relevant to predict the success rate of fully online courses, while student-content interactions are deemed as relevant to in-class lectures. The results reported in [19] confirm that learners' habits, social activities, and teamwork styles are relevant to identify the key factors influencing students performance. To deepen the analysis of the interactions between students and LMSs, in [20,21] the authors analyze the data acquired from the Moodle LMS to discover which features (e.g., total time online, number of downloads, amount of communications with peers) are significantly correlated with the final grade.…”

Generalizing Predictive Models of Admission Test Success Based on Online Interactions

“…Nevertheless, the predictive power can still sometimes be acceptable to be used. For example, the predictive power of the model trained with the same course in a previous edition was acceptable (as reported by Gitinabard et al [41]), which is useful in ensuring sustainability of the models. Nevertheless, as discussed, it is important to reuse and adapt the models whenever possible, and therefore, the process of generation of the models should be generalizable as far as possible to guarantee the scalability of the models when models cannot be transferred.…”

“…He et al [40], however, found that predictive models trained on a first edition performed well on a second edition of a MOOC. In addition, Gitinabard et al [41] analyzed the generalizability in four courses and found accurate results when transferring models, although they were better when the course was the same but in another offering. Furthermore, Hung et al [42] proposed three models to predict successful students and at-risk students and a third model to optimize the thresholds of the previous models.…”

“…This work, however, analyzes generalizability with different contexts (same/different cohort, same/different course, etc.). One of the most similar works is the article by Gitinabard et al [41]. They had two courses with two offerings each, so they could evaluate transferability in another edition and in another course.…”

Prediction of learning outcomes via clickstream data using machine learning

“…Subsequently, learners' preparation and re-Chapter 1: Introduction source usage during teaching weeks as well as their physiological and physical logs are made available as they progress through the course before their formative and summative assessment grades can be retrieved. Given the general consensus that at-risk learners respond best when interventions are provided at the earliest opportunity, such a timely process requires the predictors to be available in time for any interventions to be administered [14]. These clickstream data that are available across different courses, therefore, provide a perfect opportunity to detect unproductive learning behaviors.…”