Applying natural language processing to automatically assess student conceptual understanding from textual responses

Abstract

In this study, we applied natural language processing (NLP) techniques, within an educational environment, to evaluate their usefulness for automated assessment of students’ conceptual understanding from their short answer responses. Assessing understanding provides insight into and feedback on students’ conceptual understanding, which is often overlooked in automated grading. Students and educators benefit from automated formative assessment, especially in online education and large cohorts, by providing insights into conceptual understanding as and when required. We selected the ELECTRA-small, RoBERTa-base, XLNet-base and ALBERT-base-v2 NLP machine learning models to determine the free-text validity of students’ justification and the level of confidence in their responses. These two pieces of information provide key insights into students’ conceptual understanding and the nature of their understanding. We developed a free-text validity ensemble using high performance NLP models to assess the validity of students’ justification with accuracies ranging from 91.46% to 98.66%. In addition, we proposed a general, non-question-specific confidence-in-response model that can categorise a response as high or low confidence with accuracies ranging from 93.07% to 99.46%. With the strong performance of these models being applicable to small data sets, there is a great opportunity for educators to implement these techniques within their own classes. Implications for practice or policy: Students’ conceptual understanding can be accurately and automatically extracted from their short answer responses using NLP to assess the level and nature of their understanding. Educators and students can receive feedback on conceptual understanding as and when required through the automated assessment of conceptual understanding, without the overhead of traditional formative assessment. Educators can implement accurate automated assessment of conceptual understanding models with fewer than 100 student responses for their short response questions.

You Can Handle, You Can Teach It: Systematic Review on the Use of Extended Reality and Artificial Intelligence Technologies for Online Higher Education

“…Significant technological advances are contributing to rethinking and elevating the instructional design so that educational materials that are more specifically applicable are provided to students at their appropriate level of learning. Moreover, the publications reviewed suggest that when using key technologies, learners feel inspired to learn, retain more knowledge, and are more interested in completing the assigned activities [65,66,70,143].…”

“…Two publications focused on the automated assessment of student understanding and corrective feedback in Computer sciences [143] and the Humanities [50]. Similarly, another study presented algorithms that automatically evaluate and classify learners by groups based on their assessment using a remote laboratory system for Electric engineering [144].…”

An Automated English Essay Scoring Engine Based on Neutrosophic Ontology for Electronic Education Systems

“…The grammar, syntax, and lexical extent of the essay are all evaluated in two stages to arrive at the syntactic mark [5][6][7].…”

“…A fourth area where improvement in textual information handling is needed is in the area of ideas, semantic connections, and the surrounding information required for the notion of disambiguation. Finally, the method needs to be trustworthy and practical for educators [5][6][7].…”

Understanding English as a Foreign Language Students’ Idea Generation Strategies for Creative Writing With Natural Language Generation Tools

“…Common NLG tasks for language models include question-answering, translation and summarization (Radford et al, 2019), but in schools the application of these NLG tasks appears relatively unexplored. This is not least because, like other AI technologies, NLG educational applications such as dialogue-based tutoring systems, paraphrasing tools and chatbots are relatively new so institutions have not adopted them (Somers et al, 2021). Likewise, Zhai et al (2021) reviewed 100 AI in education studies from 2010 to 2020 and identified only three studies that might be considered NLG.…”