Automated Scoring of Scientific Creativity in German

Abstract

Automated scoring is a current hot topic in creativity research. However, most research has focused on the English language and popular verbal creative thinking tasks, such as the alternate uses task. Therefore, in this study, we provide a proof-of-concept for automated scoring of a scientific creative thinking task that assesses divergent ideation in experimental tasks in the German language. Participants are required to generate alternative explanations for an empirical observation. This work analyzed a total of 13,423 unique responses. To predict human ratings of originality, we utilized the large language model XLM-RoBERTa. The prediction model was trained on 9400 responses. In a validation dataset of 1341 responses, the correlation between predictions and ratings was strong (r = 0.81; CI-95% [0.79, 0.83]). The correlation remained strong in a held-out test set (n = 2682), with a correlation of r = 0.80; CI-95% [0.79, 0.81]. It is argued that these promising findings represent a strong proof-of-concept for automated scoring of scientific creative thinking in the German language. We encourage researchers to further investigate automated scoring of other scientific creative thinking tasks.