Improving Automated Essay Scoring by Prompt Prediction and Matching

Sun, Jingbo; Song, Tianbao; Song, Jihua; Peng, Weiming

doi:10.3390/e24091206

Cited by 3 publications

(8 citation statements)

References 25 publications

(54 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies have shown the advancements in natural language processing (NLP) and transformer-based neural language model have significantly influenced the development and effectiveness of automated essay scoring (AES) systems [6,9,13,15,18,[30][31][32][33][34][35][36][37][38].…”

Section: The Role Of Advanced Language Models In Aes Systemsmentioning

confidence: 99%

“…[18] also used BERT and RoBERTa for AES tasks. [38] proposed a multi-task learning approach for AES that outperformed the baseline system in key statistical measures. Lastly, [37] highlighted the time-consuming nature of manual essay grading and the potential of automated graders built on top of language models like BERT, RoBERTa, and DeBERTa.…”

Section: The Role Of Advanced Language Models In Aes Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Transformer-Based Neural Language Models for Writing Feedback and Automated Essay Scoring

Abraha,

Nazir

2024

Preprint

View full text Add to dashboard Cite

Writing remains a challenging skill for many students due to inadequate feedback tools. There is a need to develop more effective tools for supporting students’ writing skill development. This study aims to evaluate the effectiveness of transformer-based neural language models for assessing and automatically scoring argumentative essays written by 8th-12th grade English Language Learners (ELLs). The students’ English essays were assessed and scored based on six criteria, including cohesion, syntax, vocabulary, phraseology, grammar, and conventions. The models were trained on real teacher feedback from 2700 scored essays. We also compared various transformer-based neural language models to find the most effective model. Several metrics were used for evaluation, with the root mean square error (RMSE) as the primary measure. The results show that a specific model, DeBERTa-v3-large, outperforms others in most categories. In conclusion, this study suggests that transformer-based neural language models, especially when using the DeBERTa-v3-large model, hold significant promise in improving automated essay scoring and feedback, potentially leading to enhanced writing skills among English language learners.

show abstract

Section: The Role Of Advanced Language Models In Aes Systemsmentioning

confidence: 99%

Section: The Role Of Advanced Language Models In Aes Systemsmentioning

confidence: 99%

Evaluation of Transformer-Based Neural Language Models for Writing Feedback and Automated Essay Scoring

Abraha,

Nazir

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The corpus serves as the foundation for AEE research, so we depicted the usage of corpora in the ACEE research literature in Figure 4. Some scholars utilized large-scale exam corpora such as HSK (Hanyu Shuiping Kaoshi, the Chinese Language Proficiency Test) [18,[33][34][35][36] and MHK (Minzu Hanyu Kaoshi, Chinese Language Proficiency Test for Ethnic Minorities in China) [19,[37][38][39][40][41]. The HSK and MHK corpora contain essays written by NNS for Chinese proficiency assessments.…”

Section: Corpus Construction and Usage (Rq1)mentioning

confidence: 99%

“…This information was then used as attention information to enhance the theme relevance in the text representations generated by the DNN (deep neural network) at each layer. Sun, J. et al [34] obtained theme relevance features through two classification tasks: theme prediction and theme matching. Theme prediction involves predicting the theme that an essay belongs to from a limited set of themes, while theme matching determines whether the essay is compatible with a specific theme.…”

Section: Thematic Featuresmentioning

confidence: 99%

“…Some studies directly use the one-hot representation of the raw essay texts [50,60]. In contrast, others employ word embedding vectors pre-trained on large-scale corpora to encode essays to obtain more comprehensive semantic information [34,45]. Pretrained language models such as BERT in Chinese [34,45], HBiLSTM [45], and NEZHA [34] have provided convenience for evaluating essays using deep learning methods.…”

Section: Embedding-based Featurementioning

confidence: 99%

See 1 more Smart Citation

Automatic Essay Evaluation Technologies in Chinese Writing—A Systematic Literature Review

Yang,

He,

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Automatic essay evaluation, an essential application of natural language processing (NLP) technology in education, has been increasingly employed in writing instruction and language proficiency assessment. Because automatic Chinese Essay Evaluation (ACEE) has made some breakthroughs due to the rapid development of upstream Chinese NLP technology, many evaluation tools have been applied in teaching practice and high-risk evaluation processes. However, the development of ACEE is still in its early stages, with many technical bottlenecks and challenges. This paper systematically explores the current research status of corpus construction, feature engineering, and scoring models in ACEE through literature to provide a technical perspective for stakeholders in the ACEE research field. Literature research has shown that constructing the ACEE public corpus is insufficient and lacks an effective platform to promote the development of ACEE research. Various shallow and deep features can be extracted using statistical and NLP techniques in ACEE. However, there are still substantial limitations in extracting grammatical errors and features related to syntax and traditional Chinese Literary style. For the construction of scoring models, existing studies have shown that traditional machine learning and deep learning methods each have advantages in different corpora and feature selections. The deep learning model, which exhibits strong adaptability and multi-task joint learning potential, has broader development space regarding model scalability.

show abstract

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Xu,

Mahmud,

Lam Hoo

2024

IEEE Access

View full text Add to dashboard Cite

Artificial intelligence technology is becoming increasingly essential to education. The outbreak of COVID-19 in recent years has led many schools to launch online education. Automated online assessments have become a hot topic of interest, and an increasing number of researchers are studying Automated Essay Scoring (AES). This work seeks to summarise the characteristics of current AES systems used in English writing assessment, identify their strengths and weaknesses, and finally, analyse the limits of recent studies and research trends. Search strings were used to retrieve papers on AES systems from 2018 to 2023 from four databases, 104 of which were chosen to be potential to address the posed research aims after study selection and quality evaluation. It is concluded that the existing AES systems, although achieving good results in terms of accuracy in specific contexts, are unable to meet the needs of teachers and students in real teaching scenarios. The improvements of these systems relate to the scalability of the system for assessing different topics or styles of the essays, the accuracy of the model's predicted scores, as well as the reliability of outcomes: improving the robustness of AES models with some adversarial inputs, the richness of AES system functionality, and the development of AES assist tools.INDEX TERMS Automated essay scoring, deep learning, machine learning, natural language processing, writing assessment.

show abstract

Improving Automated Essay Scoring by Prompt Prediction and Matching

Cited by 3 publications

References 25 publications

Evaluation of Transformer-Based Neural Language Models for Writing Feedback and Automated Essay Scoring

Evaluation of Transformer-Based Neural Language Models for Writing Feedback and Automated Essay Scoring

Automatic Essay Evaluation Technologies in Chinese Writing—A Systematic Literature Review

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Contact Info

Product

Resources

About