Assessing second-language academic writing: AI vs. Human raters

Abstract

The quality of writing in a second language (L2) is one of the indicators of the level of proficiency for many college students to be eligible for departmental studies. Although certain software programs, such as Intelligent Essay Assessor or IntelliMetric, have been introduced to evaluate second-language writing quality, an overall assessment of writing proficiency is still largely achieved through trained human raters. The question that needs to be addressed today is whether generative artificial intelligence (AI) algorithms of large language models (LLMs) could facilitate and possibly replace human raters when it comes to the burdensome task of assessing student-written academic work. For this purpose, first-year college students (n=43) were given a paragraph writing task which was evaluated through the same writing criteria introduced to the generative pre-trained transformer, ChatGPT-3.5, and two human raters. The scores assigned by two human raters (Human Rater A: m=86.51, SD=13.61, range=40-100, Human Rater B: m=73.49, SD=17.98, range=10-100) revealed a statistically significant small to moderate positive correlation (r (41)=.36, p=.017). A slight but significant level of agreement was observed in the scores assigned by ChatGPT-3.5 and one of the human raters (κ=.124, p=.036). The findings suggest that reliable results could be obtained when the scores of an application and a human rater are considered and that ChatGPT may potentially assist human raters in assessing L2 college writing.