Exploring new depths: Applying machine learning for the analysis of student argumentation in chemistry

Abstract

Constructing arguments is essential in science subjects like chemistry. For example, students in organic chemistry should learn to argue about the plausibility of competing chemical reactions by including various sources of evidence and justifying the derived information with reasoning. While doing so, students face significant challenges in coherently structuring their arguments and integrating chemical concepts. For this reason, a reliable assessment of students' argumentation is critical. However, as arguments are usually presented in open‐ended tasks, scoring assessments manually is resource‐consuming and conceptually difficult. To augment human diagnostic capabilities, artificial intelligence techniques such as machine learning or natural language processing offer novel possibilities for an in‐depth analysis of students' argumentation. In this study, we extensively evaluated students' written arguments about the plausibility of competing chemical reactions based on a methodological approach called computational grounded theory. By using an unsupervised clustering technique, we sought to evaluate students' argumentation patterns in detail, providing new insights into the modes of reasoning and levels of granularity applied in students' written accounts. Based on this analysis, we developed a holistic 20‐category rubric by combining the data‐driven clusters with a theory‐driven framework to automate the analysis of the identified argumentation patterns. Pre‐trained large language models in conjunction with deep neural networks provided almost perfect machine‐human score agreement and well‐interpretable results, which underpins the potential of the applied state‐of‐the‐art deep learning techniques in analyzing students' argument complexity. The findings demonstrate an approach to combining human and computer‐based analysis in uncovering written argumentation.

Lehre in der organischen Chemie individualisieren

“…[13][14][15] Zuletzt wurden die insgesamt 1108 von Studierenden formulierten Argumente mit einem supervised ML-Modell automatisiert ausgewertet. Dieses Modell hat in englischer Sprache eine Genauigkeit von über 87 Prozent 12) und in deutscher Sprache eine Genauigkeit von über 82 Prozent. 17 tegorien gliedern sich in fokussierte Themen, Kausalität und Tiefe der Argumentation (Tabelle).…”

“…Um die individuelle Förderung bei großen Kohorten zu realisieren, haben wir auf Basis der Computational Grounded Theory 16) ein ML-Modell entwickelt, das frei formulierte Argumentationen automatisiert auswertet. 12) Dazu wurden mit unsupervised ML zunächst generalisierbare Argumentationsmuster datengetrieben aus den Antworten der Lernenden extrahiert. Die so gewonnenen Argumentationsmuster wurden mit chemiedidaktischen Interpretationsrahmen analysiert und überarbeitet.…”

Towards Effective Argumentation: Design and Implementation of a Generative Ai-Based Evaluation and Feedback System

“…Human-scored data are automatically evaluated using their developed tool, the Constructed Response Classifier (CRC) to build scoring models. The CRC is an ensemble model that concurrently harnesses eight classification algorithms (Martin et al, 2023). On the other hand, Wilson et al procured high-quality data based on credible argumentative prompts and a systematic evaluation rubric.…”

“…They then optimized the automated assessment model by applying an ensemble approach that leverages eight distinct machine learning classification models (Wilson et al, 2023). Additionally, Martin et al (2023) conducted a study that utilized data-driven clustering analysis and unsupervised learning to categorize and assess argument patterns. Their research encompasses the utilization of pre-trained large-scale language models, various deep learning techniques, and collaboration with humans in analysing students' argumentative essays (Zhai et al, 2023).…”

“…The use of LLMs in argumentation become heeded in the community of NLP and education. Many of researchers conduct a study whether the use of AI helps students to evaluate and improve their ability relevant to argumentation based on a variety of schemes (Walton, 2016;Martin et al, 2023;Spector & Mar, 2019). In particular, Toulmin's model is useful to analyse the detailed information about the argumentation and to give a visual interpretation of the logic-based argumentation (Bentahar et al, 2010;Racharak & Tojo, 2021).…”