Development of a machine learning-based tool to evaluate correct Lewis acid–base model use in written responses to open-ended formative assessment items

Abstract

Acid–base chemistry is a key reaction motif taught in postsecondary organic chemistry courses. More specifically, concepts from the Lewis acid–base model are broadly applicable to understanding mechanistic ideas such as electron density, nucleophilicity, and electrophilicity; thus, the Lewis model is fundamental to explaining an array of reaction mechanisms taught in organic chemistry. Herein, we report the development of a generalized predictive model using machine learning techniques to assess students’ written responses for the correct use of the Lewis acid–base model for a variety (N = 26) of open-ended formative assessment items. These items follow a general framework of prompts that ask: why a compound can act as (i) an acid, (ii) a base, or (iii) both an acid and a base (i.e., amphoteric)? Or, what is happening and why for aqueous proton-transfer reactions and reactions that can only be explained using the Lewis model. Our predictive scoring model was constructed from a large collection of responses (N = 8520) using a machine learning technique, i.e., support vector machine, and subsequently evaluated using a variety of validation procedures resulting in overall 84.5–88.9% accuracies. The predictive model underwent further scrutiny with a set of responses (N = 2162) from different prompts not used in model construction along with a new prompt type: non-aqueous proton-transfer reactions. Model validation with these data achieved 92.7% accuracy. Our results suggest that machine learning techniques can be used to construct generalized predictive models for the evaluation of acid–base reaction mechanisms and their properties. Links to open-access files are provided that allow instructors to conduct their own analyses on written, open-ended formative assessment items to evaluate correct Lewis model use.

Students’ Strategies, Struggles, and Successes with Mechanism Problem Solving in Organic Chemistry: A Scoping Review of the Research Literature

“…These studies emphasize that the structure of the prompt influences how well it elicits students’ responses, − aligning with studies of more traditional problem types that demonstrate how the information given to students and the framing of a problem influence the elicited reasoning. ,,, For example, in one study, first asking students to “describe in full detail what you think is happening on the molecular level” followed by asking to “explain why this reaction occurs” best elicited students’ descriptions and explanations . Further studies using constructed response items demonstrate how lexical analysis and machine learning techniques can be used to automatically analyze students’ short, written responses at scale. − These studies have been followed by research demonstrating how automated analysis of students’ responses can be employed to provide students with tutorials and adaptive interventions to support their learning; , similar research demonstrates how automated, individualized, formative feedback on multiple choice questions can similarly support students’ learning . Related approaches to constructed response items are the science writing heuristic (SWH) and writing-to-learn (WTL), which are distinct writing-focused pedagogies that have been studied in the context of writing laboratory reports and eliciting students’ descriptions and explanations of acid–base concepts, resonance, and reaction mechanisms. ,− These writing assignments involve longer student responses than constructed-response items and often include additional structures to promote opportunities for learning, such as guiding questions and reflections or peer review and revision.…”

“…For instance, contrasting cases and instructional scaffolds might be a useful tool to encourage students to consider multiple variables in active learning classrooms. ,,,,− , Furthermore, researchers suggest using problems that require students to consider mechanisms rather than memorization or that require students to apply mechanistic thinking to new situations. For example, researchers recommend constructed-response items ,− and writing assignments ,− that ask students “why” questions pertaining to reaction mechanisms or organic chemistry concepts. Engaging students in understanding why reactions occur at a deeper level could help support students in considering the multiple variables that pertain to reaction mechanisms while using their knowledge of structure–property relationships.…”

Mechanistic Reasoning in Organic Chemistry: A Scoping Review of How Students Describe and Explain Mechanisms in the Chemistry Education Research Literature

“…Of note, a majority of articles used either qualitative or mixed methods. Seven of the ten articles using quantitative methodologies have appeared mostly in recent years (between 2018 and 2021); four of these focused on machine learning methodologies to analyze students’ descriptions and explanations of reaction mechanisms. − Machine learning represents a new and growing area of quantitative research in the area of understanding students’ reasoning with reaction mechanisms. The relatively few quantitative studies indicate that leveraging quantitative methodologies to explore how students describe, explain, and reason with reaction mechanisms is an avenue for further research.…”

“…To support students’ understanding of acid–base theories in the organic chemistry context, researchers have called for scaffolding students’ reasoning with acid–base theories to support consideration of Lewis theory and the associated implicit properties such as polarity and partial charges. , Studies demonstrate that a revised general chemistry curriculum which emphasizes models-based reasoning may support students’ causal and mechanistic reasoning with Lewis acid–base theory in organic chemistry. , Other research to support students’ reasoning with acid–base theories includes writing-to-learn assignments and adaptive tutorials that use machine learning models to identify students’ use of Lewis acid–base theory in order to promote understanding of the Lewis acid–base model for proton-transfer reactions or reaction steps. − …”

Closing the gap of organic chemistry students’ performance with an adaptive scaffold for argumentation patterns

“…However, a large amount of data is required in advance (Zhou, 2016) and more adapted scaffolds need to be created to support content-related interpersonal differences (Shute and Zapata-Rivera, 2008) and acknowledge other individual differences such as motivational, metacognitive, and affective aspects, among others (Azevedo and Gasevic, 2019). An advantage of the computer-based adaptive learning system would be that the time and number of staff can be reduced and so more students have the opportunity to receive adaptive support in a shorter period of time (Dood et al, 2020;Lee et al, 2021;Yik et al, 2021). A useful application would be an automatic scoring system of students' scaffold answers with machine learning approaches as scoring is the most time-consuming part of the adaptive scaffolding process.…”