A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

Abstract

The TextEvaluator® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers and other educators select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards (CCSS). This paper provides an overview of the TextEvaluator measurement approach and summarizes evidence related to three key claims in the TextEvaluator validity argument: (a) TextEvaluator has succeeded in expanding construct coverage beyond the two dimensions of text variation that are traditionally assessed by readability metrics; (b) the TextEvaluator strategy of estimating distinct prediction models for informational, literary, and mixed texts has succeeded in generating text complexity predictions that exhibit little, if any, genre bias; and (c) TextEvaluator scores are highly correlated with text complexity judgments provided by human experts, including judgments generated via the inheritance method and judgments generated via the exemplar method. Implications with respect to the goal of helping teachers and other educators select texts that are closely aligned with the accelerated text complexity exposure trajectory outlined in the CCSS are discussed.

Helping Students Select Appropriately Challenging Text: Application to a Test of Second Language Reading Ability

“…Third, because passage complexity is evaluated using TextEvaluator, resulting estimates may be more closely aligned with cognitive theories of how readers process text and thus may be more effective at distinguishing passages that are likely to be more or less challenging for test takers (Chen & Sheehan, ; Sheehan, , ; Sheehan et al, ).…”

“…This section presents an application of the proposed reader–text matching algorithm to the problem of helping TOEFL iBT test takers select texts that are well matched to their abilities. The application is implemented with respect to the TextEvaluator ® text analysis tool, an ATCMT designed to provide text complexity scores that are closely aligned with the reading proficiency constructs targeted by many reading assessments that aid in making high‐stakes decisions, including assessments targeted at L1 readers and assessments targeted at L2 readers (Chen & Sheehan, ; Sheehan, , ; Sheehan, Kostin, Napolitano, & Flor, ).…”

Assessing Academic Reading

“…Difficult parts of texts should pose the greatest comprehension issues, demand higher "standards of coherence," and so on. While there is a wealth of studies on how to measure text complexity through automated means (Graesser et al, 2011;Graesser et al, 2014;Sheehan, 2016;Sheehan et al, 2014) and its impact on comprehension (McNamara et al, 1996;O'Reilly & McNamara, 2007;Ozuru et al, 2009;Ozuru et al, 2010), more work is needed to determine how to leverage this notion in a formal assessment context. On the one hand, a simple approach would be to report the complexity of the texts used in the assessment.…”

A Serious Game for Improving Inferencing in the Presence of Foreign Language Unknown Words

“…This was measured using the TextEvaluator tool [67]. Scores given by the tool were found to have high correlation with judgment presented by human experts [68]. Content A has 164 words while content B has 226 words.…”