Utilizing a Pretrained Language Model (BERT) to Classify Preservice Physics Teachers’ Written Reflections

Abstract

Computer-based analysis of preservice teachers’ written reflections could enable educational scholars to design personalized and scalable intervention measures to support reflective writing. Algorithms and technologies in the domain of research related to artificial intelligence have been found to be useful in many tasks related to reflective writing analytics such as classification of text segments. However, mostly shallow learning algorithms have been employed so far. This study explores to what extent deep learning approaches can improve classification performance for segments of written reflections. To do so, a pretrained language model (BERT) was utilized to classify segments of preservice physics teachers’ written reflections according to elements in a reflection-supporting model. Since BERT has been found to advance performance in many tasks, it was hypothesized to enhance classification performance for written reflections as well. We also compared the performance of BERT with other deep learning architectures and examined conditions for best performance. We found that BERT outperformed the other deep learning architectures and previously reported performances with shallow learning algorithms for classification of segments of reflective writing. BERT starts to outperform the other models when trained on about 20 to 30% of the training data. Furthermore, attribution analyses for inputs yielded insights into important features for BERT’s classification decisions. Our study indicates that pretrained language models such as BERT can boost performance for language-related tasks in educational contexts such as classification.

A machine learning-enabled open biodata resource inventory from the scientific literature

“…Both training datasets were split into 70% training, 15% validation, 15% test (hold-out) for article classification ( Table 1 ) and NER tasks ( Table 2 ). Following common practice, the training sets were used to fine-tune the models on their respective tasks, the validation sets were used to compare the fine-tuned models for selection, and the test sets were used to evaluate how the models perform on unseen data [ 23 ].…”

Extending a Pretrained Language Model (BERT) using an Ontological Perspective to Classify Students’ Scientific Expertise Level from Written Responses

“…An illustrative example of this application is evident in the work of , wherein LR models were employed to identify the causal structure present in students' scienti c explanations. Transformer-based Natural Language Processing (NLP) models, as exempli ed by prominent instances such as BERT and GPT, have become the de facto industry standard for a diverse range of NLP downstream tasks (Cochran, Cohn, Rouet, and Hastings, 2023;Wulff et al, 2023). Prior research (e.g., Cochran et al, 2022) has consistently highlighted the effectiveness of BERT-based transformers in evaluating students' responses to STEM-related questions.…”

When a machine detects student reasoning: a review of machine learning-based formative assessment of mechanistic reasoning

“…These methods require less human effort, especially when a reliable scoring rubric has already been applied to many student responses (Haudek et al, 2011). However, the mentioned natural language processing techniques are built on the simplified assumption that the word order is irrelevant to the meaning of a sentence (Wulff et al, 2022b), which complicates the detection of implicit semantic embeddings. So, traditional ML models are only sensitive to key conceptual components, which is why we define construct assessment based on shallow learning experiences as the second level of the ML-adapted ECD.…”

“…So, future research needs to check whether such cutting-edge techniques can also be used to accurately evaluate short, content-rich scientific explanations. Maybe, such technologies will produce better outcome metrics than the n-gram approach; Dood et al (2022) and Winograd et al (2021bWinograd et al ( , 2021b in chemistry education research as well as Wulff et al (2022aWulff et al ( , 2022b in physics education research have laid the foundation for future research in this area.…”