Different Approaches to Assessing the Quality of Explanations Following a Multiple-Document Inquiry Activity in Science

Abstract

This article describes several approaches to assessing student understanding using written explanations that students generate as part of a multiple-document inquiry activity on a scientific topic (global warming). The current work attempts to capture the causal structure of student explanations as a way to detect the quality of the students' mental models and understanding of the topic by combining approaches from Cognitive Science and Artificial Intelligence, and applying them to Education. First, several attributes of the explanations are explored by hand-coding and leveraging existing technologies (LSA and Coh-Metrix). Then, we describe an approach for inferring the quality of the explanations using a novel, two-phase machine learning approach for detecting causal relations and the causal chains that are present within student essays. The results demonstrate the benefits of using a machine learning approach for detecting content, but also highlight the promise of hybrid methods that combine ML, LSA and Coh-Metrix approaches for detecting student understanding. Opportunities to use automated approaches as part of Intelligent Tutoring Systems that provide feedback toward improving student explanations and understanding are discussed.

From substitution to redefinition: A framework of machine learning‐based science assessment

“…The results were used as an intervention to facilitate students' interest and retention in STEM fields. In another study, Wiley et al (2017) evaluated the quality of students' essays by investigating the causal structure of the explanations to infer students' mental models. The study provided nuanced diagnoses of students' understanding level and helped to improve student learning.…”

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

“…Other previous work has used multi-leveled coding schemes to try to characterize the quality of complete student responses (Liu et al 2016;Mao et al 2018;Wiley et al 2017). Such holistic coding schemes may be developed based on "correctness" of an explanation; however, not all such schemes may be closely tied to an underlying framework.…”

Assisting students’ writing with computer-based concept map feedback: A validation study of the CohViz feedback system

“…As the Wikipedia entries were written in German we used the assessment tool developed by Hancke et al [37], which has been designed for the German language. The system provides researchers with validated and commonly used linguistic indicators regarding the cohesion, syntactic, and lexical complexity of a text [4,5,48]. This system is comparable to Coh-Metrix, which was developed for the English language by Graesser et al [46].…”

“…Comprehensible writing can be regarded as a crucial writing skill in today's knowledge society [1][2][3]. A central feature that contributes to the comprehensibility of a text is cohesion [4,5]. Cohesion refers to grammatical or lexical devices which explicitly relate ideas within and across text segments [6].…”