ACT research report series: An argument-based approach to validation

Kane, M. T.

doi:10.1037/e426852008-001

Cited by 57 publications

(97 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…According to Kane, it is the interpretations and uses of assessments, rather than the assessments themselves, that require validation (Kane, 1992(Kane, , 2012(Kane, , 2013(Kane, , 2015. This requirement is seen to extend to formative assessments whose purpose is to support learning and teaching (Bennet, 2011).…”

Section: Argument-based Validation For Awe As Formative Assessmentmentioning

confidence: 99%

“…While some recent research has investigated AWE accuracy and usefulness separately and in different ways (Dikli & Bleyle, 2014;Lavolette, Polio, & Kahng, 2015), these studies define accuracy with reference to developer-rather than user-centric standards and leave aside the question of how accuracy problems may affect usefulness. The present research seeks to investigate these issues from the perspective of argument-based validation (Kane, 1992(Kane, , 2012(Kane, , 2013(Kane, , 2015, which provides a framework for combining potentially disparate forms of validity evidence in appraising particular interpretations or uses of an assessment. In two studies described here, we appraise evidence regarding two inferences in an interpretation/use argument for the use of AWE as a formative assessment tool in collegelevel ESL writing courses.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Ranalli

Link

Chukharev‐Hudilainen

2016

Educational Psychology

100

View full text Add to dashboard Cite

An increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggest growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses. A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level English as a Second Language (ESL) writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilisation inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities. KeywordsAcademic writing, argument-based validation, automated writing evaluation, ESL, formative assessment AWE FOR FORMATIVE ASSESSMENT 2 AbstractAn increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggests growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses (Kane, 2012). A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level ESL writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilization inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities.

show abstract

Section: Argument-based Validation For Awe As Formative Assessmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Ranalli

Link

Chukharev‐Hudilainen

2016

Educational Psychology

100

View full text Add to dashboard Cite

show abstract

“…The validity as argument framework is frequently employed when evaluating such claims (Cronbach, 1988;Kane, 1990Kane, , 2006Kane, , 2013. Kane (2013) described this framework as encompassing two steps.…”

Section: Evidence Presented In Support Of Key Claims In the Textevalumentioning

confidence: 99%

“…If any part of the argument is not plausible, however, the specified claim would be deemed invalid. Since resulting networks of inferences and supporting assumptions can become quite large quite quickly, Kane (1990) suggested focusing on those links that appear to be most open to challenge (i.e., links that appear to be "doubtful or problematic" [p. 20]). This recommendation is based on the notion that "a serious weakness in any core inference tends to undermine the argument as a whole, even if other inferences are strongly supported" (Kane, 1990, p. 13).…”

Section: Evidence Presented In Support Of Key Claims In the Textevalumentioning

confidence: 99%

A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

Sheehan

2016

ETS Research Report Series

View full text Add to dashboard Cite

The TextEvaluator® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers and other educators select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards (CCSS). This paper provides an overview of the TextEvaluator measurement approach and summarizes evidence related to three key claims in the TextEvaluator validity argument: (a) TextEvaluator has succeeded in expanding construct coverage beyond the two dimensions of text variation that are traditionally assessed by readability metrics; (b) the TextEvaluator strategy of estimating distinct prediction models for informational, literary, and mixed texts has succeeded in generating text complexity predictions that exhibit little, if any, genre bias; and (c) TextEvaluator scores are highly correlated with text complexity judgments provided by human experts, including judgments generated via the inheritance method and judgments generated via the exemplar method. Implications with respect to the goal of helping teachers and other educators select texts that are closely aligned with the accelerated text complexity exposure trajectory outlined in the CCSS are discussed.

show abstract

“…Historically, the methods of establishing a test's validity were founded on the early work of Cronbach (1971Cronbach ( , 1988Cronbach ( , 1989, Cronbach and Meehl (1955), and Kane (1992Kane ( , 1994Kane ( , 2001). In the 1940s, validity focused on the test itself and was conceptualized as a static property of an instrument.…”

Section: Evolution Of Validity As a Conceptmentioning

confidence: 99%

Construct Validity: A Unitary Concept for Occupational Therapy Assessment and Measurement

Brown

2010

Hong Kong Journal of Occupational Therapy

View full text Add to dashboard Cite

Background:In the traditional view of validity, it is divided it into three sub-types: content, criterionrelated, and construct. Validity has recently been reconceptualized as a unitary factor known as construct validity. Five sources of construct validity evidence have been specified: test content, response processes, internal structure, relationships to other variables, and consequential aspects of construct validity. They function either as general validity criteria or as standards for all assessment and measurement. Purpose: The purpose of the manuscript is to present an overview of the contemporary conceptualization of construct validity and its relevance to occupational therapy practice, education, and research. Key Issues & Implications: Taken together, the five components of construct validity evidence provide a way of addressing the multiple and interrelated validity questions that need to be answered in order to justify occupational therapy test score interpretation and use by practitioners. Given the context of evidence-based practice, professional standards of practice, increasing calls for accountability, and the fact that validity of a test, instrument, or scale is now seen as being a dynamic process, it is important for occupational therapists to be conversant with this contemporary view of construct validity, and the body of validity evidence related to the assessment tools used in clinical practice.

show abstract

ACT research report series: An argument-based approach to validation

Cited by 57 publications

References 0 publications

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

Construct Validity: A Unitary Concept for Occupational Therapy Assessment and Measurement

Contact Info

Product

Resources

About