Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education

Abstract

Developing natural language processing (NLP) models to automatically score students' written responses to science problems is critical for science education. However, collecting sufficient student responses and labeling them for training or fine-tuning NLP models is time and cost-consuming. Recent studies suggest that large-scale pre-trained language models (PLMs) can be adapted to downstream tasks without fine-tuning by using prompts. However, no research has employed such a prompt approach in science education. As students' written responses are presented with natural language, aligning the scoring procedure as the next sentence prediction task using prompts can skip the costly finetuning stage. In this study, we developed a zero-shot approach to automatically score student responses via Matching exemplars as Next Sentence Prediction (MeNSP). This approach employs no training samples. We first apply MeNSP in scoring three assessment tasks of scientific argumentation and found machine-human scoring agreements, Cohen's Kappa ranges from 0.30 to 0.57, and F1 score ranges from 0.54 to 0.81. To improve scoring performance, we extend our research to the few-shots setting, either randomly selecting labeled student responses at each grading level or manually constructing responses to fine-tune the models. We find that one task's performance is improved with more samples, Cohen's Kappa from 0.30 to 0.38, and F1 score from 0.54 to 0.59; for the two other tasks, scoring performance is not improved. We also find that randomly selected few-shots perform better than the human expert-crafted approach. This study suggests that MeNSP can yield referable automatic scoring for student-written responses while significantly reducing the cost of model training. This method can benefit low-stakes classroom assessment practices in science education. Future research should further explore the applicability of the MeNSP in different types of assessment tasks in science education and further improve the model performance.

Practical and ethical challenges of large language models in education: A systematic scoping review

“…Specifically, these LLMs, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al, 2018) and Generative Pre-trained Transformer (GPT) (Brown et al, 2020), utilise deep learning and self-attention mechanisms (Vaswani et al, 2017) to selectively attend to the different parts of input texts, depending on the focus of the current tasks, allowing the model to learn complex patterns and relationships among textual contents, such as their semantic, contextual, and syntactic relationships (Liu et al, 2023;Min et al, 2021). As several LLMs (eg, GPT-3 and Codex) have been pre-trained on massive amounts of data across multiple disciplines, they are capable of completing natural language processing tasks with little (few-shot learning) or no additional training (zero-shot learning) (Brown et al, 2020;Wu et al, 2023). This could lower the technological barriers to LLMs-based innovations as researchers and practitioners can develop new educational technologies by fine-tuning LLMs on specific educational tasks without starting from scratch (Caines et al, 2023;Sridhar et al, 2023).…”

AI and formative assessment: The train has left the station

“…The AI community has been making progress in these directions. For example, few-shot learning (Wu et al, 2023) and fine-tuned pre-trained large language models (Liu et al, 2023) are promising approaches for learning from limited data. With advances in AI interpretability, we may be able to understand how AI makes decisions and then modify them to better serve our students.…”