Agreement and Coverage of Indicators of Response to Intervention

Abstract

Purpose Agreement across methods for identifying students as inadequate responders or as learning disabled is often poor. We report (1) an empirical examination of final status (post-intervention benchmarks) and dual-discrepancy growth methods based on growth during the intervention and final status for assessing response to intervention; and (2) a statistical simulation of psychometric issues that may explain low agreement. Methods After a Tier 2 intervention, final status benchmark criteria were used to identify 104 inadequate and 85 adequate responders to intervention, with comparisons of agreement and coverage for these methods and a dual-discrepancy method. Factors affecting agreement were investigated using computer simulation to manipulate reliability, the intercorrelation between measures, cut points, normative samples, and sample size. Results Identification of inadequate responders based on individual measures showed that single measures tended not to identify many members of the pool of 104 inadequate responders. Poor to fair levels of agreement for identifying inadequate responders were apparent between pairs of measures In the simulation, comparisons across two simulated measures generated indices of agreement (kappa) that were generally low because of multiple psychometric issues inherent in any test. Conclusions Expecting excellent agreement between two correlated tests with even small amounts of unreliability may not be realistic. Assessing outcomes based on multiple measures, such as level of CBM performance and short norm-referenced assessments of fluency may improve the reliability of diagnostic decisions.

Accuracy and Validity of Methods for Identifying Learning Disabilities in a Response-to-Intervention Service Delivery Framework

“…Here, it is important to recognize that assessments of achievement by norm-referenced tests and by a CBM may be used jointly to operationalize the low achievement and instructional response components of the triangle approach to definition. In Fletcher et al (2014), no single measure was reasonably accurate in identifying the pool of inadequate responders. However, the norm-referenced assessment of untimed word-reading accuracy and a CBM assessment of oral reading fluency concurred in identifying 90/104 of the pool of inadequate responders; the 14 not identified were all average achievers and likely false positives.…”

“…However, a great deal of work remains to be done to show that assessments of growth add to the information provided by the end-point final status measure. In Fletcher et al (2014), adding the slope to the final status assessment likely added many adequate achievers to the pool of inadequate responders. Moreover, empirical studies of the value-added information of slopes have not shown that additional information is obtained.…”

“…To illustrate the underlying psychometric issues that may explain the low agreement observed in empirical studies, Fletcher et al (2014) simulated data based on two highly reliable final status measures with manipulations of reliability, the intercorrelations of the two measures, constructs measured, cut points, normative samples, and sample size. For each condition except the small sample size condition, 100,000 simulated observations were generated that recreated the condition parameters (e.g., the specified reliabilities, intercorrelations of the two measures, cut points, normative samples, and sample size for the specific condition).…”

An Experimental Evaluation of Guided Reading and Explicit Interventions for Primary-Grade Students At-Risk for Reading Difficulties

“…All students who met the criteria of a composite standard score on the Test of Word Reading Efficiency (TOWRE) or a Basic Skills Cluster standard score on the Woodcock–Johnson III Tests of Achievement (WJ III; Woodcock, McGrew, & Mather, 2001) less than 93 (i.e., below the 30th percentile) were selected for participation ( n = 218). This benchmark has been validated against external criteria as an indicator of adequate response to reading intervention at the end of Grade 1 (Fletcher et al, 2014) and used in several previous studies (Mathes et al, 1995; Torgesen, 2000). These students were randomly assigned to receive supplemental small-group intervention using a GR approach (GR group; n = 74) or an explicit instruction approach (EX group; n = 73), or the typical reading instruction and intervention provided by their schools, with no researcher-provided intervention (TSI group; n = 71).…”

Identifying Preschool Children for Higher Tiers of Language and Early Literacy Instruction Within a Response to Intervention Framework

“…We expected that preschool RTI tier-level proportions in programs would be influenced by child factors such as poverty, the language spoken at home, and special needs status (e.g., Shanahan & Lonigan, 2008). We also expected proportions would be influenced by the measures selected for use and the constructs, content, and methods that they reflected (e.g., Fletcher, Stuebing, Miciak, & Denton, 2012). This information about proportions of children identified for greater instructional intensity is fundamental for programs seeking to address critical policy questions about feasibility and the resources, staff, time, and cost needed to implement RTI.…”