Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Abstract

An increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggest growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses. A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level English as a Second Language (ESL) writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilisation inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities. KeywordsAcademic writing, argument-based validation, automated writing evaluation, ESL, formative assessment AWE FOR FORMATIVE ASSESSMENT 2 AbstractAn increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggests growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses (Kane, 2012). A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level ESL writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilization inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities.

Technology and Feedback Design

“…First developed in the 1960s, automated systems for assessing student writing have primarily been used to score student work (Link, Dursun, Karakaya, & Hegelheimer, 2014). The last decade has seen the emergence of automated writing evaluation (AWE) tools which not only assess writing, but provide students with formative feedback on language components such as grammar and structure (Chapelle, Cotos, & Lee, 2015;Link et al, 2014;Ranalli, Link, & Chukharev-Hudilainen, 2017). Feedback generated by AWE systems is instant and specific to individual student submissions, and generally focuses on diagnosing sentence-level errors in language mechanisms.…”

“…However, AWE tools aimed at providing feedback on discourse characteristics, such as components of an introduction, have also been developed (Chapelle et al, 2015). Recent research relating to AWE tools has largely emerged from language disciplines, particularly English as a second or foreign language, and indeed marketing of AWE tools has increasingly targeted language disciplines (Bai & Hu, 2017;Ranalli et al, 2017). It is suggested that AWE tools can support educators by providing feedback on sentence mechanics, enabling educators to address higher-level writing components such as content and audience awareness (Ranalli et al, 2017).…”

A Left Turn: Automated Feedback and Activity Generation for Student Writers

“…While there is evidence suggesting that existing tools can be helpful [12][13][14][15][16][17][18], this paper describes a demo that takes a left turn. Specifically, this paper presents a novel idea for writing feedback with revision activities.…”

Addressing Diversity in CALL Evaluation through Arguments and Theory-of-Action

“…A follow-on investigation to this study conducted by myself and two colleagues (Ranalli, Link, and Chukharev-Hudilainen 2016) shows how explicit arguments can serve the purpose of decision-making and help in communicating among stakeholders. I undertook this study in my dual capacity as both a researcher and a coordinator for the ESL writing program in which the Chapelle, Cotos, and Lee (2015) study was conducted.…”