Testing the Impact of Novel Assessment Sources and Machine Learning Methods on Predictive Outcome Modeling in Undergraduate Biology

Abstract

High levels of attrition characterize undergraduate science courses in the USA. Predictive analytics research seeks to build models that identify at-risk students and suggest interventions that enhance student success. This study examines whether incorporating a novel assessment type (concept inventories [CI]) and using machine learning (ML) methods (1) improves prediction quality, (2) reduces the time point of successful prediction, and (3) suggests more actionable course-level interventions. A corpus of university and course-level assessment and non-assessment variables (53 variables in total) from 3225 students (over six semesters) was gathered. Five ML methods were employed (two individuals, three ensembles) at three time points (pre-course, week 3, week 6) to quantify predictive efficacy. Inclusion of course-specific CI data along with university-specific corpora significantly improved prediction performance. Ensemble ML methods, in particular the generalized linear model with elastic net (GLMNET), yielded significantly higher area under the curve (AUC) values compared with non-ensemble techniques. Logistic regression achieved the poorest prediction performance and consistently underperformed. Surprisingly, increasing corpus size (i.e., amount of historical data) did not meaningfully impact prediction success. We discuss the roles that novel assessment types and ML techniques may play in advancing predictive learning analytics and addressing attrition in undergraduate science education.

Applying machine learning to automatically assess scientific models

“…ML has the potential to revolutionize science assessments by significantly improving the functionality and automaticity of scoring open‐ended and drawn responses, as they are able to target complex constructs that traditional assessments cannot evaluate (Zhai, 2021). Prior studies have shown the substantial potential of ML to make evidentiary inference based on large‐scale and complex data (e.g., Bertolini et al, 2021; Rosenberg & Krist, 2021), which are difficult to analyze using traditional statistical methods. These studies suggest that ML has the potential to improve the functionality of assessments to make accurate decisions based on evidentiary data and rigorous inference.…”

The Implementation of Different Types of Rubrics in a Hong Kong Secondary School

“…Last but not least, understanding how the students make use of rubrics as their tools for selfregulation perhaps would be a very important issue (Efklides, 2011), other studies stressed the importance of high transparency, which help students to identify an accurate learning goal, which in turn foster students to take responsibility for self-regulation (Bertolini et al, 2021;de Boer et al, 2021;Hasselquist & Bertolini, 2018). Furthermore, the use of rubrics in learning assessment can minimize the teacher's overly subjective practice in assessing and reduce students' academic dishonesty behavior (Surahman & Wang, 2022).…”

Practices and Theories: How Can Machine Learning Assist in Innovative Assessment Practices in Science Education

“…Because of this feature, ML could help educators identify and interpret patterns in big datasets. For example, Bertolini et al (2021) employed a dataset with 53 variables from 3225 undergraduate students to develop algorithms to predict students' attrition. It is anticipated that once we have such models, educators may use the models to predict students' attrition and then develop better strategies to support those students who may have a high risk of attrition.…”