Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

Abstract

This paper describes HASbot, an automated text scoring and real‐time feedback system designed to support student revision of scientific arguments. Students submit open‐ended text responses to explain how their data support claims and how the limitations of their data affect the uncertainty of their explanations. HASbot automatically scores these text responses and returns the scores with feedback to students. Data were collected from 343 middle‐ and high‐school students taught by nine teachers across seven states in the United States. A mixed methods design was applied to investigate (a) how students’ utilization of HASbot impacted their development of uncertainty‐infused scientific arguments; (b) how students used feedback to revise their arguments, and (c) how the current design of HASbot supported or hindered students’ revisions. Paired sample t tests indicate that students made significant gains from pretest to posttest in uncertainty‐infused scientific argumentation, ES = 1.52 SD, p < 0.001. Linear regression analysis results indicate that students' HASbot use significantly contributed to their posttest performance on uncertainty‐infused scientific argumentation while gender, English language learner status, and prior computer experience did not. From the analysis of videos, we identified several affordances and limitations of HASbot.

Managing uncertainty in scientific argumentation

“…In Lee et al's () learning progression analysis of a context‐based questionnaire, students were most uncertain about the strength of evidence in support of a claim and about scientific reasoning between theory and evidence. Students who were more aware of the degree of their uncertainty were able to engage in higher levels of argumentation, using warrants and justification to connect theory and evidence (Lee et al, ). In a similar vein, Buck et al () revealed that students tended to be uncertain about what data should be included in evidence to shape a coherent argument.…”

“…Argumentation, as a core practice of science, is a dialogic practice permeated with uncertainty (Buck, Lee, & Flores, ; Lee et al, , ; Noroozi, Weinberger, Biemans, Mulder, & Chizari, ; Osborne & Patterson, ). Uncertainty in argumentation is derived from opposition and controversy in which individuals holding different, conflict, or opposing arguments to debate, discuss, and evaluate each other's ideas (Leitão, ).…”

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

“…These efforts identified key components of the practice of argumentation (e.g., claim) via automated scoring. Such scores were then used as feedback to individual students to help in revising their arguments (Lee et al 2019), thereby demonstrating the potential of automated scoring systems. Finally, recent efforts have attempted to create automated scoring systems that align with cognitive models of learning, such as LPs (Anderson et al 2018).…”

Maschinelles Lernen mit Aussagen zur Modellkompetenz

“…1 illustriert das grundsätzliche Potenzial offener Aufgabenformate, ein breites Spektrum auch komplexerer Kognitionen zu erfassen. Da Modellkompetenz bisher in der fachdidaktischen Forschung vorwiegend mit geschlossenen Formaten erfasst wird (Mathesius und Krell 2019;Nicolaou und Constantinou 2014), eröffnen Verfahren des ML die Option, auch Antworten auf Fragen im offenen Format zeiteffizient auszuwerten (Lee et al 2019;Liu et al 2014;Moharreri et al 2014).…”

“…Andere Studien mit ML von Interviewdaten oder schriftlichen Antworten auf offene Fragestellungen befassen sich mit dem naturwissenschaftlichen Erklären (Linn et al 2014) und Argumentieren (Zhu et al 2017). Diese Bereiche sind theoretisch gut fundiert und es liegen Sammlungen von Begründungen oder Erklärungen in längeren mündlichen oder schriftlichen Ausführungen vor (Lee et al 2019); zum Beispiel im Inhaltsbereich Jahreszeiten, K hier auch unter dem Blick auf Schülervorstellungen (Dam und Kaufmann 2008). In Liu et al (2016) werden elf Studien mit ML zitiert.…”