Validity arguments for diagnostic assessment using automated writing evaluation

Abstract

Two examples demonstrate an argument-based approach to validation of diagnostic assessment using automated writing evaluation (AWE). Criterion ® , was developed by Educational Testing Service to analyze students' papers grammatically, providing sentence-level error feedback. An interpretive argument was developed for its use as part of the diagnostic assessment process in undergraduate university English for academic purposes (EAP) classes. The Intelligent Academic Discourse Evaluator (IADE) was developed for use in graduate EAP university classes, where the goal was to help students improve their discipline-specific writing. The validation for each was designed to support claims about the intended purposes of the assessments. We present the interpretive argument for each and show some of the data that have been gathered as backing for the respective validity arguments, which include the range of inferences that one would make in claiming validity of the interpretations, uses, and consequences of diagnostic AWE-based assessments. New diagnostic assessments of writing with automated writing evaluation (AWE) promise individualized diagnostic feedback aimed at guiding students' revision and raising their consciousness about specific aspects of their writing to help them learn. However, like any assessment, diagnostic assessments need to be evaluated in view of the validity of their intended interpretations, uses and consequences. Despite the need for validity arguments, the evaluation of AWE has tended to focus on accuracy of the system (Dikli, 2006), even if other approaches to validation have been recognized (Bennett and Bejar, 1998; Yang, Buckendahl, Juszkiewicz, & Bhola, 2002;Cotos & Pendar, 2008;Powers, Burstein, Chodorow, Fowles, & Kukich, 2001; Yang et al. 2002). In particular, Clauser, Kane and Swanson (2002) conceptualized how to frame a validity argument by identifying five types of inferences that one might wish to make on the basis of results from an assessment with automated scoring. This paper demonstrates how such inferences can be used to develop validity arguments for two AWE-based diagnostic assessments. This paper illustrates 1) the claims that users of AWE systems want to be able to make about the value of the classroom assessment and the data used to support such claims, as well as 2) the need for inferences and claims to be developed to reflect the goals of the AWE system and its use. We provide two examples to demonstrate that the variety of approaches taken to AWE-based diagnostic assessment can be evaluated through the use of an argument-based validity framework. The first system, Criterion®, was developed by Educational Testing Service to analyze students' papers grammatically and provide sentence-level error feedback. This system was used as part of the diagnostic assessment in five undergraduate university English for academic purposes (EAP) classes. The second one, Intelligent Academic Discourse Evaluator (IADE), was developed for use in six graduate EAP university classes, wher...

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

“…Although Kane's argument-based approach does not define the methods necessary for evaluating inferences, the validity argument is extremely valuable as a systematic and evidence-based approach (Aryadoust, 2013). Thus, the present study provides evidence for two inferences and a set of assumptions that fit within a more extensive framework proposed by Chapelle, Cotos, and Lee (2015) in order to investigate the use of automated writing evaluation for formative assessment of L2 writing. Chapelle et al, (2015) presented their IUA based on seven inferences that would require backing to support the integration of Criterion into a college-level ESL writing course at Iowa State University.…”

“…Thus, the present study provides evidence for two inferences and a set of assumptions that fit within a more extensive framework proposed by Chapelle, Cotos, and Lee (2015) in order to investigate the use of automated writing evaluation for formative assessment of L2 writing. Chapelle et al, (2015) presented their IUA based on seven inferences that would require backing to support the integration of Criterion into a college-level ESL writing course at Iowa State University. Five of their inferences (evaluation, generalization, explanation, extrapolation, and utilization) are outlined in Clauser et al, (2002), who conceptualized how to frame a validity argument on the basis of results from an automated scoring assessment.…”

“…Five of their inferences (evaluation, generalization, explanation, extrapolation, and utilization) are outlined in Clauser et al, (2002), who conceptualized how to frame a validity argument on the basis of results from an automated scoring assessment. In addition to those five inferences, Chapelle et al, (2015) included two additional inferences: ramification, which is critical for making the claim that learning results from assessment use, and domain definition, which is important for defining the real-world domain of interest (see also Chapelle, 2011;Chapelle, Enright, & Jamieson, 2008). The authors of the present study are associated with Iowa State University and its ESL writing program; the first author is currently a course coordinator who decides what materials and tools will be incorporated into the curriculum.…”

“…In their interpretation-use argument, Chapelle et al (2015) do not define usefulness other than to say that useful Criterion feedback allows students to make decisions about revisions. By contrast, classroom-based studies of AWE have defined the usefulness of AWE feedback in terms of clarity, specificity, and relevance, as well as the types of features that AWE feedback can address.…”

“…In a study of Criterion use in grades 6-12 involving thousands of students (both English L2 learners and English native speakers) across the U.S., more than 70% of 33,171 essay submissions were submitted only once for feedback, demonstrating that "... most students did not exploit the revision capabilities of the Criterion system" (Attali, 2004, p. 4). A study of Criterion-based revisions made by university-level ESL writing students (Lee, Li, & Hegelheimer, 2012; see also Chapelle et al, 2015) documented six types of response to Criterion feedback: no change, remove, add, delete, change, and transpose. In more than 50% of cases, students made no changes, which the authors attributed to frequently inaccurate feedback.…”