Generating Automated Text Complexity Classifications That Are Aligned With Targeted Text Complexity Standards

Sheehan, Kathleen M.; Kostin, Irene; Futagi, Yoko; Flor, Michael

doi:10.1002/j.2333-8504.2010.tb02235.x

Cited by 43 publications

(53 citation statements)

References 30 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The dimensions obtained in the most recent factor analysis in this line of research (Sheehan et al, 2010) involve the features and dimensions shown in Table 1.…”

Section: Modeling Writing Development and Writing Constructs: An Apprmentioning

confidence: 99%

Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency

Deane

Quinlan

Kostin

2011

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.Key words: writing assessment, CBAL, automated scoring ii (2008) Context and PurposeIn practice, assessment design involves complex tradeoffs. For instance, many testing programs assess writing directly by administering generic prompts under strict time limits. This type of assessment has gained in popularity relative to indirect measures of writing skill in response to the concern that writing should be assessed by actually requiring students to write, but it has been criticized as encouraging simplistic, formulaic approaches to instruction that fail to develop the complex skills students need to learn in order to become effective writers (Hillocks, 1987(Hillocks, , 2002. Portfolio-based writing assessment may encourage richer, more appropriate forms of instruction, but it raises issues of reliability and validity. Automated scoring methods have been introduced and advanced as instructional aids, but they have been criticized as focusing attention too much on mechanical correction of errors rather than encouraging critical engagement with content. In other words, it is no simple matter to design summative assessments that are truly tests worth teaching to (Shepard, 2002). Success in creating tests that truly encourage learning will require a complex, sustained effort that takes into account an extraordinary range of elements, including at least the following:• the cognitive and instructional literatures that define what students learn and how they best learn it• psychometric constraints on effective assessment• institutional and public policy constraints that ultimately determine test feasibility ETS has recently instituted a research initiative, Cognitively Based Assessments of, for, and as Learning (CBAL), intended to address this problem and create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math (see Bennett & Gitomer, 2009). This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning. The role of automated scoring for CBAL cannot be discussed, however, without an in-depth presentation of the CBAL initiative.This presentation occupies section 1.2, after which section 1.3 examines the role of automated scoring within this philosophy. The remaining sections of th...

show abstract

“…The dimensions obtained in the most recent factor analysis in this line of research (Sheehan et al, 2010) involve the features and dimensions shown in Table 1.…”

Section: Modeling Writing Development and Writing Constructs: An Apprmentioning

confidence: 99%

Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency

Deane

Quinlan

Kostin

2011

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because many important complexity features are known to function differently within texts from different genres (Hiebert, 2012;Hiebert & Mesmer, 2013a, 2013bSheehan, 2013;Sheehan, Flor, & Napolitano, 2013;Sheehan, Kostin, Futagi, & Flor, 2010), three distinct prediction models are estimated: one optimized for application to informational texts, one optimized for application to literary texts, and one optimized for application to mixed texts (i.e., texts that incorporate a mixture of informational and literary elements). 2 Additional information about each step is summarized below.…”

Section: An Overview Of the Approachmentioning

confidence: 99%

“…Methods for addressing this problem are discussed in a number of recent papers (see e.g., Deane, Sheehan, Sabatini, Futagi, & Kostin, 2006;Graesser, McNamara, & Kulikowich, 2011;Sheehan, Kostin, & Futagi, 2007;Sheehan et al, 2010). In each case, a two-step solution is proposed.…”

Section: Step 2: Define Construct-relevant Text Featuresmentioning

confidence: 99%

A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

Sheehan

2016

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

The TextEvaluator® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers and other educators select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards (CCSS). This paper provides an overview of the TextEvaluator measurement approach and summarizes evidence related to three key claims in the TextEvaluator validity argument: (a) TextEvaluator has succeeded in expanding construct coverage beyond the two dimensions of text variation that are traditionally assessed by readability metrics; (b) the TextEvaluator strategy of estimating distinct prediction models for informational, literary, and mixed texts has succeeded in generating text complexity predictions that exhibit little, if any, genre bias; and (c) TextEvaluator scores are highly correlated with text complexity judgments provided by human experts, including judgments generated via the inheritance method and judgments generated via the exemplar method. Implications with respect to the goal of helping teachers and other educators select texts that are closely aligned with the accelerated text complexity exposure trajectory outlined in the CCSS are discussed.

show abstract

“…These texts serve as exemplars for the level of reading ability at a given grade level. This corpus was introduced as an evaluation corpus for readability models in the recent past (Sheehan et al, 2010;Nelson et al, 2012;Flor et al, 2013), so we used it to compare our model with other systems.…”

Section: Corporamentioning

confidence: 99%

Assessing the relative reading level of sentence pairs for text simplification

Vajjala¹,

Meurers²

2014

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

While the automatic analysis of the readability of texts has a long history, the use of readability assessment for text simplification has received only little attention so far. In this paper, we explore readability models for identifying differences in the reading levels of simplified and unsimplified versions of sentences.Our experiments show that a relative ranking is preferable to an absolute binary one and that the accuracy of identifying relative simplification depends on the initial reading level of the unsimplified version. The approach is particularly successful in classifying the relative reading level of harder sentences.In terms of practical relevance, the approach promises to be useful for identifying particularly relevant targets for simplification and to evaluate simplifications given specific readability constraints.

show abstract

Generating Automated Text Complexity Classifications That Are Aligned With Targeted Text Complexity Standards

Cited by 43 publications

References 30 publications

Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency

Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency

A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

Assessing the relative reading level of sentence pairs for text simplification

Contact Info

Product

Resources

About