Using machine learning to identify the most at-risk students in physics classes

Abstract

Machine learning algorithms have recently been used to predict students' performance in an introductory physics class. The prediction model classified students as those likely to receive an A or B or students likely to receive a grade of C, D, F or withdraw from the class. Early prediction could better allow the direction of educational interventions and the allocation of educational resources. However, the performance metrics used in that study become unreliable when used to classify whether a student would receive an A, B, or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the class (the DFW outcome) because the outcome is substantially unbalanced with between 10% to 20% of the students receiving a D, F, or W. This work presents techniques to adjust the prediction models and alternate model performance metrics more appropriate for unbalanced outcome variables. These techniques were applied to three samples drawn from introductory mechanics classes at two institutions (N ¼ 7184, 1683, and 926). Applying the same methods as the earlier study produced a classifier that was very inaccurate, classifying only 16% of the DFW cases correctly; tuning the model increased the DFW classification accuracy to 43%. Using a combination of institutional and in-class data improved DFW accuracy to 53% by the second week of class. As in the prior study, demographic variables such as gender, underrepresented minority status, first-generation college student status, and low socioeconomic status were not important variables in the final prediction models.

Physics Teachers’ Perceptions about Their Judgments within Differentiated Learning Environments: A Case for the Implementation of Technology

“…We recommend recent developments of artificial intelligence (AI) to address limitations discovered in teachers' judgments. One of the fields of AI study, machine learning (ML), is a predictive model that has recently received great attention in the assessment of physics learning [19][20][21]69,70]. The implementation of ML studies for educational purposes, namely educational data mining (EDM) and learning analytics (LA), could lead to employing ML models to develop predictive systems to monitor students' learning.…”

“…The scaffolding approach assists students' learning, through a self-determined pace, to plan, to monitor, and to measure the extent to which they have been progressing in their physics learning. Diversity in physics has been promoted through other studies, including the adaptive tutoring system (ATS) [16,17], the computerized adaptive test (CAT) [18], and employing machine learning (ML) algorithms [19][20][21].…”

Physics Teachers’ Perceptions about their Judgment within Differentiated Learning Environment: An Implication for Technology Implementation

“…We recommends recent development of artificial intelligence (AI) to address limitations discovered in teachers' judgment. One of the fields of AI study, namely machine learning (ML), is a predictive model that has recently become novel attention in the assessment of physics learning [19][20][21]47,48]. Implementation of ML studies for educational purposes, namely educational data mining (EDM) and learning analytics (LA) employing ML models to develop predictive system to monitor students' learning.…”

“…Scaffolding approach assists students' learning through self-determination pace to plan, to monitor, and to measure the extent to which they have been progressing in their physics learning. Diversity in physics has been promoted through other studies including adaptive tutoring system [16,17], computerized adaptive test [18], and employing machine learning algorithms [19][20][21].…”

An application of Bayesian inference to examine student retention and attrition in the STEM classroom

“…There is increasing interest in examining how combining these traditional formative data types with course-specific data-driven tools and assessment data (e.g., LMS usage patterns, diagnostic tests) extracted from intelligent systems may differentially inform models suitable for course-level instructor actions in the STEM classroom. These novel assessment types, in conjunction with academic characteristics and personalized data records, have been shown to improve the overall performance of ML algorithms (Lee et al, 2015;Zabriskie et al, 2019;Yang et al, 2020;Zhai et al, 2020a,b;Bertolini et al, 2021aBertolini et al, ,b, 2022. However, frequentist and non-Bayesian methods have been the primary techniques utilized in these analyses to assess competing performance variability between different algorithms and to identify the significant features that drive overall ML model performance.…”