The effectiveness of learning analytics for identifying at-risk students in higher education

Peering into the crystal ball of the disengaged: What happens to students that do not submit an early assessment item?

“…This supports data from earlier pilot studies that indicated that a high proportion of students are still updating their subject enrolments in Weeks 1 and 2 of session (Linden & Webster, 2019), leading to difficulty in identifying students who are genuinely disengaging. Literature shows the efficacy of 'at-risk' models to accurately determine student disengagement with only two weeks of data is very poor (Kuzilek et al, 2015;Foster & Siddle, 2019). It is therefore important not to target campaigns too early.…”

“…One constant throughout these studies is the difficulty in modelling the myriad interactions between learners and learning design as well as the challenge, possibly insurmountable, in finding a one-size-fits all model to apply at scale. Approaching from a complex systems perspective can lead to promising results by applying simple rules that take into account a carefully chosen mix of data (Foster & Siddle, 2019).…”

Widening the net to reduce the debt: Reducing student debt by increasing identification of completely disengaged students

“…Over the past decade, universities have tried several different approaches to designing at risk models. Broadly speaking, models attempt to balance the demographic, academic and activity data of students to some degree, however, very different approaches exist concerning model complexity (Wolff et al, 2014;Kuzilek et al, 2015;Jayaprakash et al, 2014;Foster & Siddle, 2019;Akçapınar et al, 2019). Despite years of research and development of a complex model at The Open University, the predictive capacity was far from perfect, especially when attempting to make predictions early in the session (Kuzilek et al, 2015).…”

“…Another approach is to acknowledge the inherent complexities of the system in predicting student retention (Forsman et al, 2014) and use a simplified model. Once again however, the predictive capacity of simplified models is modest at best at early points in the session, and improves over the duration of a session (Foster & Siddle, 2019). This presents an even greater challenge given that the decision to defer a student must be made prior to census date, with limited time and therefore data to feed into the model.…”

“…Absence from the LMS for 2 weeks included a large number of students (n=22) who would still go on to pass at least one subject. Using learning analytics at scale to exclusively identify disengaged students at risk of complete failure seems to be comparably difficult to predicting students at risk of some failure (Wolff et al, 2014;Kuzilek et al, 2015;Jayaprakash et al, 2014;Foster & Siddle, 2019;Akçapınar et al, 2019). When looking at the subjects and courses of these students however, it was clear that certain subjects were overrepresented in the students who were identified as completely disengaged, but went on to pass (false positives).…”