Potential of Automated Writing Evaluation Feedback

Abstract

This paper presents an empirical evaluation of automated writing evaluation (AWE) feedback used for L2 academic writing teaching and learning. It introduces the Intelligent Academic Discourse Evaluator (IADE), a new web-based AWE program that analyzes the introduction section to research articles and generates immediate, individualized, and discipline-specific feedback. The purpose of the study was to investigate the potential of IADE's feedback. A mixed-methods approach with a concurrent transformative strategy was employed. Quantitative data consisted of responses to Likert-scale, yes/no, and open-ended survey questions; automated and human scores for first and final drafts; and pre-/posttest scores. Qualitative data contained students' first and final drafts as well as transcripts of think-aloud protocols and Camtasia computer screen recordings, observations, and semistructured interviews. The findings indicate that IADE's colorcoded and numerical feedback possesses potential for facilitating language learning, a claim supported by evidence of focus on discourse form, noticing of negative evidence, improved rhetorical quality of writing, and increased learning gains.

Computer‐Assisted Language Testing

“…Research on automated writing evaluation has resulted in products such as Intelligent Essay Assessor (Pearson), e-rater (ETS), and IntelliMetric ® (Vantage Learning) that are capable of analyzing lexical measures, syntax, and discourse structure of essays. The Intelligent Academic Discourse Evaluator (IADE) is another example of a Web-based AWE program that utilizes NLP techniques to provide feedback at the level of rhetorical functions in research writing (Cotos, 2011). IADE has become the prototype of a complex AWE system currently under development at Iowa State University.…”

“…According to Cotos (2011), supporters suggest that AWE systems are generally in close agreement with human raters and are thus more time-and cost-effective. They may also foster learner autonomy, promote the process writing approach that involves writing multiple drafts, and lead to individualized assessment.…”

“…Critics, however, claim that the use of such systems encourages students to focus on surface features such as grammar and vocabulary rather than meaning. In addition, automated assessment of essays diminishes the role of instructors and impels students to adjust their writing to the evaluation criteria of these systems (Cotos, 2011).…”

SOS français : Conception et évaluation d’un didacticiel d’aide à la rédaction interactif

“…Ainsi, Cotos (2011) et Sauro (2009 soulignent les effets positifs du feedback correctif fourni par l'ordinateur sur le développement de certaines compétences langagières.…”

Furthering and applying move/step constructs: Technology-driven marshalling of Swalesian genre theory for EAP pedagogy

“…Cotos (2009) took a step further, developing a genre-based automated writing evaluation (AWE) program called the Intelligent Automated Discourse Evaluator (IADE). This tool, grounded in second language and skill acquisition theories, translates the results of automated move analysis to move-level feedback, facilitating students' focus on the functional meaning of the RA Introduction discourse, learning of Introduction conventions, and improvement of research writing quality (Cotos, 2011(Cotos, , 2012(Cotos, , 2014.…”