Exploring the potential of using an AI language model for automated essay scoring

Mizumoto, Atsushi; Eguchi, Masaki

doi:10.1016/j.rmal.2023.100050

Cited by 126 publications

(37 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, we still do not know the degree to which each higher education institution accepts generative AI. Then researchers are required to rate it from guidelines, but the process of human rating of texts can be fraught with issues that fatigue, subjectivity, and inconsistency may produce unreliable results (Hussein et al, 2019;Mizumoto & Eguchi, 2023). Generative AI provides a faster, more consistent alternative, potentially transforming rating method, as the reliability of the ChatGPT rating was higher than the reliability of the human rating in this study.…”

Section: Practical Implicationsmentioning

confidence: 84%

“…Those students were also split fairly evenly in their preferences between generative AI and human feedback, showing that each form of feedback has its own perceived bene ts. Further, another study investigates the use of generative AI for automated essay scoring in assessing TOEFL essays (Mizumoto & Eguchi, 2023). Mizumoto and Eguchi (2023) compared AI-generated scores to human benchmarks and explored the effect of incorporating linguistic features on scoring accuracy.…”

Section: Evaluating Acceptance Of Generative Ai: Generative Ai and Hu...mentioning

confidence: 99%

See 1 more Smart Citation

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Ioku,

Kondo,

Watanabe

2024

Preprint

View full text Add to dashboard Cite

This study investigates the performance of generative artificial intelligence (AI) in evaluating the acceptance of generative AI technologies within higher education guidelines, reflecting on the implications for educational policy and practice. Drawing on a dataset of guidelines from top-ranked universities, we compared generative AI evaluations with human evaluations, focusing on acceptance, performance expectancy, facilitating conditions, and perceived risk. Our study revealed a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI, suggesting that generative AI can accurately reflect human judgment in this context. Further, we found positive associations between ChatGPT-rated acceptance and performance expectancy and facilitating conditions, while a negative correlation with perceived risk. These results validate generative AI evaluation, which also extends the application of the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology framework from individual to institutional perspectives.

show abstract

Section: Practical Implicationsmentioning

confidence: 84%

Section: Evaluating Acceptance Of Generative Ai: Generative Ai and Hu...mentioning

confidence: 99%

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Ioku,

Kondo,

Watanabe

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Other studies [26, 27] have extended the use of neural networks by employing transformer‐based pretrained language models for AES, demonstrating their effectiveness. The capabilities of GPT‐3.5, particularly in AES and grading, were explored in a recent study [28].…”

Section: Related Workmentioning

confidence: 99%

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

Cho,

Huang,

Kwon

2024

ETRI Journal

View full text Add to dashboard Cite

As automated essay scoring (AES) has progressed from handcrafted techniques to deep learning, holistic scoring capabilities have merged. However, specific trait assessment remains a challenge because of the limited depth of earlier methods in modeling dual assessments for holistic and multi‐trait tasks. To overcome this challenge, we explore providing comprehensive feedback while modeling the interconnections between holistic and trait representations. We introduce the DualBERT‐Trans‐CNN model, which combines transformer‐based representations with a novel dual‐scale bidirectional encoder representations from transformers (BERT) encoding approach at the document‐level. By explicitly leveraging multi‐trait representations in a multi‐task learning (MTL) framework, our DualBERT‐Trans‐CNN emphasizes the interrelation between holistic and trait‐based score predictions, aiming for improved accuracy. For validation, we conducted extensive tests on the ASAP++ and TOEFL11 datasets. Against models of the same MTL setting, ours showed a 2.0% increase in its holistic score. Additionally, compared with single‐task learning (STL) models, ours demonstrated a 3.6% enhancement in average multi‐trait performance on the ASAP++ dataset.

show abstract

“…Research has also examined automated essay assessment tools with AI for feedback and grading. Mizumoto and Eguchi (2023) evaluated the two approaches to assess the effectiveness of AI and human grading on student writing. The reliability of AI grading as a formative assessment is evident from the high correlation between grades and human grades.…”

Section: Higher Education Perceptions Of Ai Toolsmentioning

confidence: 99%

The Impact of Artificial Intelligence Tools on Academic Writing Instruction in Higher Education: A Systematic Review

Aljuaid

2024

Preprint

View full text Add to dashboard Cite

With the growth of Artificial Intelligence technologies, there is interest in studying their potential impact on university academic writing courses. This study examined whether AI tools are replacing these courses by exploring how they effectively replace traditional academic writing instruction and this shift’s potential benefits and drawbacks. The researcher reviewed existing literature on integrating AI tools into academic writing instruction. The findings provide insights to educators navigating the integration of Artificial Intelligence tools into writing curricula while maintaining instructional quality and academic integrity standards. By synthesizing the latest research, this study can inform decisions about the appropriate use of Artificial Intelligence in teaching essential writing skills. Increased use of Artificial Intelligence writing tools has sparked debate about their role in academic writing instruction. Universities like Stanford have updated policies around Artificial Intelligence tool usage and academic integrity. The University of California issued guidance acknowledging the prevalence of generative Artificial Intelligence on campuses. Middlebury College banned classroom use of ChatGPT over concerns it could impede critical thinking and writing skill development. Results show that while Artificial Intelligence helps with grammar and style, questions remain about its impact on creativity and critical thinking. However, Artificial Intelligence is not replacing university writing courses. These courses teach critical thinking, research, citation, argumentation, creativity, originality, and ethics, which Artificial Intelligence lacks. Academic writing courses offer a complete learning experience. Artificial Intelligence may improve academic writing but is unlikely to replace traditional courses soon. A balanced approach integrating Artificial Intelligence support while preserving core elements of academic writing education appears most effective for preparing students for diverse writing challenges.

show abstract

Exploring the potential of using an AI language model for automated essay scoring

Cited by 126 publications

References 45 publications

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

The Impact of Artificial Intelligence Tools on Academic Writing Instruction in Higher Education: A Systematic Review

Contact Info

Product

Resources

About