Semi‐automatic coding of open‐ended text responses in large‐scale assessments

Abstract

Background In the context of large‐scale educational assessments, the effort required to code open‐ended text responses is considerably more expensive and time‐consuming than the evaluation of multiple‐choice responses because it requires trained personnel and long manual coding sessions. Aim Our semi‐supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses. Method We map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets. Results With an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%. Conclusion Combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort.

Towards adaptive support for self‐regulated learning of causal relations: Evaluating four Dutch word vector models

“…(2014) and Andersen et al. (2023) reported reductions in teachers' workload and coding efforts of 60% and 52%, respectively, on average. Both studies targeted German language, and our study shows the potential and promise for Dutch language as well.…”

“…In this sense, this work adds to previous approaches to reduce coding or assessment efforts. For instance, using clustering approaches for the semi-automated coding of openended text responses, Horbach et al (2014) and Andersen et al (2023) reported reductions in teachers' workload and coding efforts of 60% and 52%, respectively, on average. Both studies targeted German language, and our study shows the potential and promise for Dutch language as well.…”

Automatic Code Generation for Android Applications Based on Improved Pix2code

“…The code generation process of Pix2code is Figure 7. And since the sigmoid is always less than 1, there will be no gradient explosion problem [24][25][26]. However, LSTM finds it difficult to predict the current word based on sequence information when understanding code patterns.…”

To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses

“…To learn which cluster corresponds to which score, the model still requires labeled training data. Otherwise, the cluster structure can be used to enhance assessment in other ways; for example, for classroom engagement (Paiva et al., 2014) or for assisted scoring, where raters inspect clusters for assigning scores to response groups (Andersen et al., in print; Basu et al., 2013; Wolska et al., 2014). Clustering can also be combined with supervised learning in various ways, such as automatically scoring clustered responses (Andersen & Zehner, 2021) or to only use labeled cluster centroids for training (Zesch et al., 2015).…”