Applying Natural Language Processing Techniques to an Assessment of Student Conceptual Understanding

Abstract

This work-in-progress briefly surveys a selection of open-source Natural Language Processing (NLP) tools and investigates their utility to the qualitative researcher. These NLP tools are widely used in the field of lexical analysis, which is concerned with automating the generation of useful information from human language using a variety of machine processes. Recent research shows that the statistical analysis of software recognized linguistic features can benchmark certain mental processes, such as cognitive load. This investigation generates those linguistic indicators using transcripts from a multi-year, interview based study and compares them to a qualitative analysis of a subject's conceptual understanding of various engineering topics. Our intermediary findings indicate a correlation between changes in the linguistic indicators introduced in this paper and a qualitatively coded analysis of conceptual understanding over time. Future work will involve increasing the breadth of the dataset to further establish the fidelity of this approach and expand on the premise of using automatically generated linguistic indicators to aid the qualitative researcher.

Using Natural Language Processing to Facilitate Student Feedback Analysis

“…In engineering education, there have been limited applications of NLP on either the research or teaching side. The more modern applications have applied standard statistical and machine learning techniques such as rule-based classifiers for assessing student responses [19]- [21], college mission statements [22], writing exercises [23], and emotions in student stories of their transitions to university [24]. Unfortunately, these kinds of rule-based systems tend to be brittle and poorly handle variations in language to express the same concept.…”