A Comprehensive Review of Automated Essay Scoring (AES) Research and Development

Lim, Chun Then; Bong, Chih How; Wong, Wee Sian; Lee, Nung Kion

doi:10.47836/pjst.29.3.27

Cited by 10 publications

(3 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lim et al developed three supervised generic architectures for AES: content similarity, ML, and hybrid approaches [9]. They introduced a novel framework for assessing essays based on content and linguistic characteristics, and proposed five evaluation methods to assess the effectiveness of AES models, advocating for the use of quadratic weighted kappa (QWK) as a common technique.…”

Section: Related Workmentioning

confidence: 99%

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Xu,

Mahmud,

Lam Hoo

2024

IEEE Access

View full text Add to dashboard Cite

Artificial intelligence technology is becoming increasingly essential to education. The outbreak of COVID-19 in recent years has led many schools to launch online education. Automated online assessments have become a hot topic of interest, and an increasing number of researchers are studying Automated Essay Scoring (AES). This work seeks to summarise the characteristics of current AES systems used in English writing assessment, identify their strengths and weaknesses, and finally, analyse the limits of recent studies and research trends. Search strings were used to retrieve papers on AES systems from 2018 to 2023 from four databases, 104 of which were chosen to be potential to address the posed research aims after study selection and quality evaluation. It is concluded that the existing AES systems, although achieving good results in terms of accuracy in specific contexts, are unable to meet the needs of teachers and students in real teaching scenarios. The improvements of these systems relate to the scalability of the system for assessing different topics or styles of the essays, the accuracy of the model's predicted scores, as well as the reliability of outcomes: improving the robustness of AES models with some adversarial inputs, the richness of AES system functionality, and the development of AES assist tools.INDEX TERMS Automated essay scoring, deep learning, machine learning, natural language processing, writing assessment.

show abstract

Section: Related Workmentioning

confidence: 99%

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Xu,

Mahmud,

Lam Hoo

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…however, its widespread and comprehensive implementation has not yet occurred (Machicao, 2019). The main goal of implementing ASES in educational fields and institutions is to reduce the time, costs, and other resources required for assessment and eliminate reliability problems associated with manual assessments carried out by teachers (Lim et al, 2021).…”

Section: Digital Assessment Model Asesmentioning

confidence: 99%

Analysis of Needs for a Digital Automated Short Essay Scoring (Ases) Based Assessment Model in English Learning in Vocational Secondary Schools

Susilawati,

Khaira,

Rambe

et al. 2023

LANGLIT

View full text Add to dashboard Cite

The aim of this research is to look at the needs analysis of the digital-based assessment model Automated Short Essay Scoring (ASES) in English language learning in Vocational High Schools. This research is a research and development (RD) research carried out in the Computer Network Technology Department at SMK Negeri 1 Percut Sei Tuan. This research is included in mixed methods research with the assessment development model used, namely the Borg and Gall model which consists of ten stages,: needs analysis, development plan, initial product development, initial test or one-on-one test, initial product revision, field test, product revision, field implementation test (operational test), revising the final product, and implementation in the field. This research focuses on the first step of all the steps in developing the Borg and Gall model above, namely needs analysis activities. The results of this research show that the implementation of digital assessment models needs to be carried out in English language learning. The use of ASES digital-based assessment media is an urgent need for assessment to be implemented to take into account the knowledge, skills, dispositions and attitudes needed to equip students to keep up with the increasingly changing digital world. Furthermore, identification of the development of the ASES digital-based assessment model in English language learning at Vocational High Schools was carried out by conducting FGDs with teachers and students of SMK Negeri 1 Percut Sei Tuan resulting in an agreement that in learning English students need to be equipped with digital skills.

show abstract

“…All these AES applications are reported to correlate with human rater scores to a certain extent (Gierl et al, 2014) even though they may fail to detect off-topic or plagiarized essays, which human raters have developed a critical eye for (Hoang, 2011). According to Lim et al (2021) and Ifenthaler (2022) the most widely used AES tools are listed as PEG, IEA, e-rater and IntelliMetric.…”

Section: Introductionmentioning

confidence: 99%

Assessing second-language academic writing: AI vs. Human raters

GECKİN,

KIZILTAŞ,

ÇINAR

2023

Journal of Educational Technology and Online Learning

View full text Add to dashboard Cite

The quality of writing in a second language (L2) is one of the indicators of the level of proficiency for many college students to be eligible for departmental studies. Although certain software programs, such as Intelligent Essay Assessor or IntelliMetric, have been introduced to evaluate second-language writing quality, an overall assessment of writing proficiency is still largely achieved through trained human raters. The question that needs to be addressed today is whether generative artificial intelligence (AI) algorithms of large language models (LLMs) could facilitate and possibly replace human raters when it comes to the burdensome task of assessing student-written academic work. For this purpose, first-year college students (n=43) were given a paragraph writing task which was evaluated through the same writing criteria introduced to the generative pre-trained transformer, ChatGPT-3.5, and two human raters. The scores assigned by two human raters (Human Rater A: m=86.51, SD=13.61, range=40-100, Human Rater B: m=73.49, SD=17.98, range=10-100) revealed a statistically significant small to moderate positive correlation (r (41)=.36, p=.017). A slight but significant level of agreement was observed in the scores assigned by ChatGPT-3.5 and one of the human raters (κ=.124, p=.036). The findings suggest that reliable results could be obtained when the scores of an application and a human rater are considered and that ChatGPT may potentially assist human raters in assessing L2 college writing.

show abstract

A Comprehensive Review of Automated Essay Scoring (AES) Research and Development

Cited by 10 publications

References 46 publications

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

A Systematic Literature Review: Are Automated Essay Scoring Systems Competent in Real-Life Education Scenarios?

Analysis of Needs for a Digital Automated Short Essay Scoring (Ases) Based Assessment Model in English Learning in Vocational Secondary Schools

Assessing second-language academic writing: AI vs. Human raters

Contact Info

Product

Resources

About