Using Neural Tensor Networks for Open Ended Short Answer Assessment

Gautam, Deepak; Rus, Vasile

doi:10.1007/978-3-030-52237-7_16

Cited by 6 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Assessment Of Short Constructed Responsesmentioning

confidence: 99%

“…Maharjan et al (2018) and Uto and Uchida (2020) applied LSTM networks (Hochreiter & Schmidhuber, 1997) to the task. Gautam and Rus (2020) evaluated systems based on neural tensor networks, which incorporate structured data from knowledge graphs to improve predictions for the dataset from Maharjan et al (2018). T ransformer language models such as BERT (Devlin et al, 2019) were successfully applied to the task, too, and could be used to achieve the latest state‐of‐the‐art results for the SemEval‐2013 data (Camus & Filighera, 2020; Poulton & Eliens, 2021; Sung et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Coding energy knowledge in constructed responses with explainableNLPmodels

Gombert

Mitri

Karademir

et al. 2022

Computer Assisted Learning

View full text Add to dashboard Cite

Background Formative assessments are needed to enable monitoring how student knowledge develops throughout a unit. Constructed response items which require learners to formulate their own free‐text responses are well suited for testing their active knowledge. However, assessing such constructed responses in an automated fashion is a complex task and requires the application of natural language processing methodology. In this article, we implement and evaluate multiple machine learning models for coding energy knowledge in free‐text responses of German K‐12 students to items in formative science assessments which were conducted during synchronous online learning sessions. Dataset The dataset we collected for this purpose consists of German constructed responses from 38 different items dealing with aspects of energy such as manifestation and transformation. The units and items were implemented with the help of project‐based pedagogy and evidence‐centered design, and the responses were coded for seven core ideas concerning the manifestation and transformation of energy. The data was collected from students in seventh, eighth and ninth grade. Methodology We train various transformer‐ and feature‐based models and compare their ability to recognize the respective ideas in students' writing. Moreover, as domain knowledge and its development can be formally modeled through knowledge networks, we evaluate how well the detection of the ideas within responses translated into accurate co‐occurrence‐based knowledge networks. Finally, in terms of the descriptive accuracy of our models, we inspect what features played a role for which prediction outcome and if the models pick up on undesired shortcuts. In addition to this, we analyze how much the models match human coders in what evidence within responses they consider important for their coding decisions. Results A model based on a modified GBERT‐large can achieve the overall most promising results, although descriptive accuracy varies much more than predictive accuracy for the different ideas assessed. For reasons of comparability, we also evaluate the same machine learning architecture using the SciEntsBank 3‐Way benchmark with an English RoBERTa‐large model, where it achieves state‐of‐the‐art results in two out of three evaluation categories.

show abstract

Section: Assessment Of Short Constructed Responsesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Coding energy knowledge in constructed responses with explainableNLPmodels

Gombert

Mitri

Karademir

et al. 2022

Computer Assisted Learning

View full text Add to dashboard Cite

show abstract

“…The best model achieved a testing accuracy of 0.760 and a Cohen's Kappa statistic of 0.684. In [13], they suggested an approach to represent reference and student answers with a graph and to represent concepts using built-in vectors, learned from the knowledge graph directly. The indirect relationships between concepts, the vectors encoded, and a knowledge graph is created by extracting the concepts and their superficial relationships from the reference answers.…”

mentioning

confidence: 99%

Short Answers Assessment Approach based on Semantic Network

Mijbel

Sadiq

2022

eijs

View full text Add to dashboard Cite

Finding similarities in texts is important in many areas such as information retrieval, automated article scoring, and short answer categorization. Evaluating short answers is not an easy task due to differences in natural language. Methods for calculating the similarity between texts depend on semantic or grammatical aspects. This paper discusses a method for evaluating short answers using semantic networks to represent the typical (correct) answer and students' answers. The semantic network of nodes and relationships represents the text (answers). Moreover, grammatical aspects are found by measuring the similarity of parts of speech between the answers. In addition, finding hierarchical relationships between nodes in networks. The similarity is then calculated, and students' answers are evaluated. The best results are for weights (α = 0.1, = 0.6, γ = 0.3) = 1.82 from 5, giving more weight in nodes similarity by 6, least similarity by relationships by 3, and least similarity by parts of speech by 1.

show abstract