A self-regulated learning analytics prediction-and-intervention design: Detecting and supporting struggling biology students.

Abstract

We investigated the effects of a learning analytics-driven prediction modeling platform and a brief digital self-regulated learning skill training program targeted to support undergraduate biology students identified as likely to perform poorly in the course. A prediction model comprising prior knowledge scores and learning management system log data of student activities during the first 2 weeks in the course was applied to flag students who were likely to earn a C or worse (N = 143). Students who were flagged were randomized into a flagged treatment (N = 79) or flagged control (N = 64) condition. We found that training students who were flagged as likely to perform poorly significantly improved their achievement on unit exams, compared with students who were also flagged but did not receive the training. The effect of training on final examination was mediated by unit exam achievement. In addition, the students who were predicted to perform well (N = 83) and flagged treatment groups did not differ statistically significantly on academic performance. Training also had a significant effect on final course performance with students in the flagged treatment and nonflagged groups outperforming the flagged control students. The results indicate that an algorithm that uses behavioral data to predict achievement does so with sufficient accuracy to detect the large differences in achievement earned by two groups of learners distinguishable by their early, digital learning behaviors, and that a brief $15-minute digital skills training was sufficient to ameliorate these achievement differences when deployed before the first unit exam. Educational Impact and Implications StatementWhen STEM (science, technology, engineering, and mathematics) undergraduates access resources like readings, notes, study guides, and quizzes on the course sites that accompany their lectures, they produce digital traces of learning events. These learning events can be interpreted based on the cognitive and metacognitive processes that the resource affords, serve as proxies for students' internal self-regulated learning processes, and be used to predict the achievement self-regulated learning is known to produce. We used machine learning to build an algorithm based on these digital events that predicted students' exam performance, then provided feedback and training or an additional biology learning activity to those who were predicted to perform poorly, and who were open to receiving learning support. Those who spent around 15 minutes completing a digital, "Science of Learning to Learn" training performed better on exams than students who were predicted to perform poorly but were not asked to complete training and no worse on exams than those who were predicted to perform well. This suggests that the data students produce when learning online can be leveraged to identify those likely to struggle and brief digital training can close the achievement gap predicted by that early pattern of behavior.

Leveraging complexity frameworks to refine theories of engagement: Advancing self‐regulated learning in the age of artificial intelligence

“…Network‐based approaches to deriving features from log data were useful for predicting student outcomes. Moreover, the metrics associated with the networks can be interpreted from a self‐regulation perspective that has practical implications for artificial intelligence efforts in learning analytics, as well as for interrogating how the digital tools designed to improve SRL and found to benefit academic achievement might achieve such effects (eg, Bernacki et al, 2020; Cogliano et al, 2022). Static representations of dynamic processes derived from the markov approaches utilized in this study (ie, transitivity calculated for probability transitions), or other statistical approaches that rely upon time intensive data to capture dynamic learning processes (Asparouhov et al, 2018), are extremely useful for advancing theory, predicting student success and examining the outcome of interventions.…”

“…Students were classified 1 into control ( n = 48), treatment ( n = 95) and not flagged ( n = 83) using a combination of a pretest score and counts of resources accessed in the course which were submitted to a forward selection logistic regression with a 10‐fold cross‐validation process. Full description of the prediction model approach is outside the scope of the current study and but can be found elsewhere (Cogliano et al, 2022).…”

“…Log data provide an electronic history of digital traces of student interactions (ie, clicks) with course content (Bernacki, 2018). Log data can be used for various analytic applications such as prediction modelling (Cogliano et al, 2022), recommender systems (Fazeli et al, 2017) and data dashboards (Klerkx et al, 2017) that provide instructors and students with information about learners' engagement with digital content. Within the area of prediction modelling, logs of student click events within an LMS are often used to examine frequency of access to particular pieces of course content or elements of the LMS.…”

Investigating bifactor modeling of biology undergraduates’ task values and achievement goals across semesters.

“…More recent syntheses have documented the promise of providing support for SRL practices (Zheng, 2016), as well as evidence of the effects of training in such practices (Broadbent & Poon, 2015;Theobald, 2021). A class of digital, SRL interventions has been developed to target self-regulatory skills that are particularly useful to students who pursue learning objectives involving declarative and conceptual knowledge acquisition as demanded in STEM coursework (Bernacki et al, 2021;Bernacki, Vosicka, et al, 2020;Cogliano et al, 2021Cogliano et al, , 2022. These brief, digital interventions can be delivered directly within the digital platforms where hybrid STEM courses house learning materials and activities, and they have been found to improve adoption of desirable learning behaviors (Bernacki, Vosicka, et al, 2020) and exam and course performance for learners (Bernacki et al, 2021;Bernacki, Vosicka, et al, 2020).…”

Predicting student outcomes using digital logs of learning behaviors: Review, current standards, and suggestions for future work

“…There has been growing interest in the timely identification of students who are likely to perform poorly in for-credit Science, Technology, Engineering, and Math (STEM) classes. Once identified, interventions attempting to prevent attrition can be made (Pistilli, Willis, & Campbell, 2014;Pritchard & Wilson, 2003;Zajacova, Lynch, & Espenshade, 2005;Cogliano, Bernacki, Hilpert, & Strong, 2022). Identifying students is particularly important in STEM disciplines given the high attrition rates of students typically underrepresented in these fields (National Academies of Sciences Engineering & Medicine, 2016) and the threats to the supply of qualified STEM professionals that attrition brings (Dai & Cromley, 2014).…”