TDNN: A Two-stage Deep Neural Network for Prompt-independent Automated Essay Scoring

Jin, Cancan; He, Ben; Hui, Kai; Sun, Le

doi:10.18653/v1/p18-1100

Cited by 69 publications

(62 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Relatively few researchers have made progress on generic essay scoring: Phandi et al (2015) introduces a Bayesian regression approach that extracts N -gram features then capitalizes on correlated features across prompts. Jin et al (2018) shows promising prompt-independent results using an LSTM architecture with surface and part-of-speech N -gram inputs, underperforming prompt-specific models by relatively small margins across all ASAP datasets. But in implementations, much of the work of practitioners is based on workarounds for prompt-specific models; , for instance, describes psychometric techniques for measuring generic writing ability across a small sample of known prompts.…”

Section: Domain Transfermentioning

confidence: 99%

Should You Fine-Tune BERT for Automated Essay Scoring?

Mayfield¹,

Black²

2020

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

Most natural language processing research now recommends large Transformer-based models with fine-tuning for supervised classification tasks; older strategies like bag-ofwords features and linear models have fallen out of favor. Here we investigate whether, in automated essay scoring (AES) research, deep neural models are an appropriate technological choice. We find that fine-tuning BERT produces similar performance to classical models at significant additional cost. We argue that while state-of-the-art strategies do match existing best results, they come with opportunity costs in computational resources. We conclude with a review of promising areas for research on student essays where the unique characteristics of Transformers may provide benefits over classical methods to justify the costs.

show abstract

Section: Domain Transfermentioning

confidence: 99%

Should You Fine-Tune BERT for Automated Essay Scoring?

Mayfield¹,

Black²

2020

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

show abstract

“…We also want to emphasize that extraction of a massive number of features vs. 23 features adds to time complexity as well. We also compared our system to work published recently in 2018; the system is called TDNN [87] which uses a two-layer neural network to reach an average QWK score of 0.7365, i.e., 7.1% lesser than us. To the best of our knowledge, our system uses the minimum number of features compared to existing systems with better results.…”

Section: G Results and Discussionmentioning

confidence: 99%

Syntactic, Semantic and Sentiment Analysis: The Joint Effect on Automated Essay Evaluation

Janda

Pawar

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Manual grading of essays by humans is time-consuming and likely to be susceptible to inconsistencies and inaccuracies. In recent years, an abundance of research has been done to automate essay evaluation processes, yet little has been done to take into consideration the syntax, semantic coherence and sentiments of the essay's text together. Our proposed system incorporates not just the rule-based grammar and surface level coherence check but also includes the semantic similarity of the sentences. We propose to use Graph-based relationships within the essay's content and polarity of opinion expressions. Semantic similarity is determined between each statement of the essay to form these Graph-based spatial relationships and novel features are obtained from it. Our algorithm uses 23 salient features with high predictive power, which is less than the current systems while considering every aspect to cover the dimensions that a human grader focuses on. Fewer features help us get rid of the redundancies of the data so that the predictions are based on more representative features and are robust to noisy data. The prediction of the scores is done with neural networks using the data released by the ASAP competition held by Kaggle. The resulting agreement between human grader's score and the system's prediction is measured using Quadratic Weighted Kappa (QWK). Our system produces a QWK of 0.793. INDEX TERMS Natural language processing, semantic analysis, sentiment analysis and text mining, automated essay evaluation.

show abstract

“…Relatively few researchers have made progress on generic essay scoring: Phandi et al (2015) introduces a Bayesian regression approach that extracts N -gram features then capitalizes on correlated features across prompts. Jin et al (2018) shows promising prompt-independent results using an LSTM architecture with surface and part-of-speech N -gram inputs, underperforming prompt-specific models by relatively small margins across all ASAP datasets. But in implementations, much of the work of practitioners is based on workarounds for prompt-specific models; Wilson et al (2019), for instance, describes psychometric techniques for measuring generic writing ability across a small sample of known prompts.…”

Section: Domain Transfermentioning

confidence: 99%

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

2020

View full text Add to dashboard Cite

Readability assessment aims to automatically classify text by the level appropriate for learning readers. Traditional approaches to this task utilize a variety of linguistically motivated features paired with simple machine learning models. More recent methods have improved performance by discarding these features and utilizing deep learning models. However, it is unknown whether augmenting deep learning models with linguistically motivated features would improve performance further. This paper combines these two approaches with the goal of improving overall model performance and addressing this question. Evaluating on two large readability corpora, we find that, given sufficient training data, augmenting deep learning models with linguistically motivated features does not improve state-of-the-art performance. Our results provide preliminary evidence for the hypothesis that the state-of-theart deep learning models represent linguistic features of the text related to readability. Future research on the nature of representations formed in these models can shed light on the learned features and their relations to linguistically motivated ones hypothesized in traditional approaches.

show abstract

TDNN: A Two-stage Deep Neural Network for Prompt-independent Automated Essay Scoring

Cited by 69 publications

References 24 publications

Should You Fine-Tune BERT for Automated Essay Scoring?

Should You Fine-Tune BERT for Automated Essay Scoring?

Syntactic, Semantic and Sentiment Analysis: The Joint Effect on Automated Essay Evaluation

Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications

Contact Info

Product

Resources

About