Automated Writing Analysis for Writing Pedagogy

Abstract

This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers. Automated Writing Analysis for writing pedagogy: From healthy tension to tangible prospects Elena Cotos AbstractThis article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers.

How to provide automated feedback on the writing process? A participatory approach to design writing analytics tools

“…AESs are grading systems that can be used for summative assessment, to replace or assist teachers in assessing writing quality (Dikli, 2006), for example e-rater (Attali & Burstein, 2006). By contrast, AWEs are intended as formative assessment tools, providing more detailed feedback and correction suggestions (Cotos, 2015), for example Criterion (Link, Dursun, Karakaya, & Hegelheimer, 2014) and AWA/AcaWriter (Knight, Shibani, Abel, Gibson, & Ryan, 2020, Knight, Martinez-Maldonado, Gibson, & Buckingham Shum, 2017. ITSs are the most complex systems.…”

“…All three types of systems have been extensively studied in the writing context and have been shown to enhance student motivation, autonomy, and improve writing quality to different extents (Cotos, 2015). However, the majority of these systems use a product-oriented approach, in which feedback is provided on students' written products (Cotos, 2015;Wang, Shang, & Briody, 2013). Only some tools provide additional resources to aid the writing process.…”

“…Indeed, automated, personalized writing tools are generally hard to develop for writing, for at least two reasons. First, as writing is an illdefined domain (Allen et al, 2015;Steenbergen-Hu & Cooper, 2014), writing specialists need to be involved in the development of such tools (Cotos, 2015). Second, these systems are used less and less effective if they are not integrated into instructors' learning design (i.e.…”

Studying disciplinary corpora to teach the craft of Discussion

“…iii Moodle, or Modular Object-Oriented Dynamic Learning Environments, is an open-source course management system that offers customizable features to tailor online learning environments (https://moodle.org). iv In a subsequent version of the course, the students have been using a move/step concordancer, which is built in an online AWE application for research writing based on our annotated corpus data (Cotos, 2015).…”

Discourse Classification into Rhetorical Functions for AWE Feedback

“…The amount of empirical evidence suggesting their effectiveness and potential benefits for language learners is accumulating (Attali, 2004;Chen & Cheng, 2008;Cotos, 2014) and is, therefore, motivating the development of new intelligent writing applications. An example of such applications is the Research Writing Tutor (RWT), a scale-up from its IADE prototype which generates discourse-level feedback on all RA sections (Cotos, 2015). In this paper, we present only part of a bigger development project and focus on the ML approach employed to build the Introductions analysis and feedback engine of RWT.…”