When a machine detects student reasoning: a review of machine learning-based formative assessment of mechanistic reasoning

Martin, Paul Peter; Graulich, Nicole

doi:10.1039/d2rp00287f

Cited by 15 publications

(14 citation statements)

References 132 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…ML has also been recognised for its potential to achieve a deeper analysis of student work with recent articles published in CERP providing inspiration and guidance for teachers. Two articles that appear in this current issue include a review of studies where ML has been used to assess mechanistic reasoning in organic chemistry (Martin and Graulich, 2023) and an application of ML in assessment of student explanations (Frost et al, 2023). Rubrics and frameworks have been shared (Raker et al, 2023;Yik et al, 2023) that can be further built upon in future studies.…”

Section: Current Considerations In Assessment Of Student Learning In ...mentioning

confidence: 99%

Establishing a delicate balance in the relationship between artificial intelligence and authentic assessment in student learning

Lawrie

2023

Chem. Educ. Res. Pract.

View full text Add to dashboard Cite

Across the past few decades, a gamut of technologies has emerged and been adopted as part of enabling student learning. These technologies and digital tools have been explored in terms of their affordances and their limitations during implementation in teaching practices. Teachers have actively worked to balance how a technology serves as a vehicle for learning against the challenges that are introduced through its implementation. In recent years, due to our increased reliance on digital tools and online learning environments, our education communities have first railed against, and then rallied for, each appearance of a new website, tool or platform. Whilst initial reactions can be negative (such as that recently observed on the appearance of the artificial intelligence (AI) based chatbot tool ChatGPT), many teachers will progress towards adoption of technologies in their practices once the affordances have been teased out. In some ways, as an analogy, teaching practice could be considered as an equilibrium reaction responding to the pressures of change. In this Editorial, I recognise that teachers are adaptive and creative therefore research that evidences authentic practice using AI to support student learning will increase. In parallel, as a journal, we are facing new challenges relating to the role of AI in authorship and peer review, most publishers are grappling with establishing their position on any authorship that involves AI generated text.

show abstract

Section: Current Considerations In Assessment Of Student Learning In ...mentioning

confidence: 99%

Establishing a delicate balance in the relationship between artificial intelligence and authentic assessment in student learning

Lawrie

2023

Chem. Educ. Res. Pract.

View full text Add to dashboard Cite

show abstract

“…36 The automated analysis of mechanistic reasoning features in student responses to WTL assignments represents just one of several approaches in the literature to characterize students' mechanistic reasoning through ML. 45 One of the applications of automated evaluation with ML are adaptive interventions and automated feedback for student responses to open-ended tasks. 23 For example, students could be provided with adaptive tutorials for specific concepts based on the automated score they receive for a related constructed response item.…”

Section: The Role Of Technology In Evaluating Written Explanationsmentioning

confidence: 99%

“…24 These technologies show promise for the future of engaging students in complex, open-ended tasks by addressing the practical limitation of instructors not having the time or resources to read and evaluate responses from multiple students. 45 However, the same technology used for many of the ML applications in STEM education can also be used for natural language generation, as seen with the recent introduction of generative AI chatbots. 25 Generative AI chatbots work by utilizing large language models (LLMs), which are trained on large amounts of publicly available text (including text available on the Internet) to understand and produce natural language.…”

Section: The Role Of Technology In Evaluating Written Explanationsmentioning

confidence: 99%

“…For example, students could be provided with adaptive tutorials for specific concepts based on the automated score they receive for a related constructed response item. , Additionally, research demonstrates the translation of ML models to develop an automated, formative feedback tool for the organic chemistry WTL assignments specifically . These technologies show promise for the future of engaging students in complex, open-ended tasks by addressing the practical limitation of instructors not having the time or resources to read and evaluate responses from multiple students . However, the same technology used for many of the ML applications in STEM education can also be used for natural language generation, as seen with the recent introduction of generative AI chatbots .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments

Watts,

Dood,

Shultz

et al. 2023

J. Chem. Educ.

View full text Add to dashboard Cite

Chemistry education research demonstrates the value of open-ended writing tasks, such as writing-to-learn (WTL) assignments, for supporting students’ learning with topics including reasoning about reaction mechanisms. The emergence of generative artificial intelligence (AI) technology, such as chatbots ChatGPT and Bard, raises concerns regarding the value of open-ended writing tasks in the classroom; one concern involves academic integrity and whether students will use these chatbots to produce sufficient responses to open-ended writing tasks. The present study investigates the degree to which generative AI chatbots exhibit mechanistic reasoning in response to organic chemistry WTL assignments. We produced responses from three generative AI chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) to two WTL assignments developed to elicit students’ mechanistic reasoning. Using previously reported machine learning models for analyzing student writing in response to the WTL assignments, we analyzed the chatbot responses for the inclusion of features pertinent to mechanistic reasoning. Herein, we report quantitative analyses of (1) the differences between chatbot responses on the two assignments and (2) the differences between chatbot and authentic student responses. Findings indicate that chatbots respond differently to different WTL assignments. Additionally, the chatbots rarely incorporated the discussion of electron movement, a key feature of mechanistic reasoning. Furthermore, the chatbots, in general, do not engage in mechanistic reasoning at the same level as students. We contextualize the results by considering academic integrity with the assumption that students’ intentions are to engage in academically honest behavior, and we focus on understanding the ethical uses of generative AI for classroom assignments.

show abstract

“…Learners need timely feedback, especially during formative stages of learning. , One way to provide such feedback is through automated text scoring approaches that are built using natural language processing and machine learning techniques. − Many text analysis tools have been developed for chemistry assessments. ,− However, several barriers remain for implementing these computer-assisted assessment technologies into instructional environments; barriers include time costs, software costs, and lack of technological support or training . To increase the uptake of these tools, the technologies must be automated, simple, and easily accessible by the user (i.e., an instructor)…”

Section: Purposementioning

confidence: 99%

Implementation of an R Shiny App for Instructors: An Automated Text Analysis Formative Assessment Tool for Evaluating Lewis Acid–Base Model Use

2023

View full text Add to dashboard Cite

Acid−base chemistry, and in particular the Lewis acid−base model, is foundational to understanding mechanistic ideas. This is due to the similarity in language chemists use to describe Lewis acid−base reactions and nucleophile− electrophile interactions. The development of artificial intelligence and machine learning technologies has led to the creation of predictive text analysis models that evaluate a large number of open-ended, written formative assessment items. One of these machine learning-based tools developed by the authors evaluates correct Lewis acid−base model use. Bridging the gap between educational research, technological innovation, and instructional practice, we report the development of a web-based, interactive app using R Shiny application technologies that automates scoring of written assessments about acid−base chemistry. Results given by this Shiny app, in the form of on-screen output or a downloadable file, provide instructors with immediate feedback to evaluate acid−base instruction in their organic chemistry courses.

show abstract

When a machine detects student reasoning: a review of machine learning-based formative assessment of mechanistic reasoning

Abstract: In chemistry, reasoning about the underlying mechanisms of observed phenomena lies at the core of scientific practices. The process of uncovering, analyzing, and interpreting mechanisms for explanations and predictions requires...

Cited by 15 publications

References 132 publications

Establishing a delicate balance in the relationship between artificial intelligence and authentic assessment in student learning

Establishing a delicate balance in the relationship between artificial intelligence and authentic assessment in student learning

Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments

Implementation of an R Shiny App for Instructors: An Automated Text Analysis Formative Assessment Tool for Evaluating Lewis Acid–Base Model Use

Contact Info

Product

Resources

About