Evaluating Screening Procedures Across Changes to the Statewide Achievement Test

Abstract

Effective universal screening within a multitiered system of support requires psychometrically adequate and efficient screening procedures that allow educators to predict which students are most likely to need additional support. Educators in upper elementary and middle school grades are often interested in predicting proficiency on statewide achievement tests to identify students needing remedial support (Espin, Wallace, Lembke, Campbell, & Long, 2010; Fuchs, Fuchs, & Compton, 2010). Several states have changed their statewide achievement tests over the past 5 years, and these changes may pose some challenges for conducting effective universal screening in upper elementary and middle school grades. Little attention has been devoted to examining how changes to the state achievement test may affect the decision-making accuracy of screening procedures. The purpose of this study was to examine the relative stability of decision-making accuracy estimates across a change in the state achievement test. Evaluating Screening Measures Previous research on screening measures can be roughly divided into studies that evaluated predictive validity and studies that also evaluated decision-making (i.e., 747390A EIXXX10.

Co-occurrence of academic and behavioral risk within elementary schools: Implications for universal screening practices.

“…For English/ language arts, 28.1% of students were proficient and 70.4% were not. Such an approach to dichotomization is concordant with previous research, including diagnostic accuracy studies examin-ing statewide achievement test scores as binary outcomes (e.g., Klingbeil, Van Norman, Nelson, & Birr, 2018).…”

Evaluating aimswebPlus math as a universal screening measure in upper elementary and middle school.

“…This may overestimate the obtained sensitivity and specificity values (Jenkins et al, 2007), particularly if the local cut-scores were applied to future cohorts of students in this district. There is some evidence that locally derived cut-scores (applied to measures with good to excellent AUC values) result in relatively stable estimates across subsequent cohorts (Klingbeil et al, 2018; Nelson et al, 2017), but we are not advocating that the local cut-scores used in this study be adopted in other districts. Instead, we suggest that the differences between the sensitivities obtained using the vendor-recommended cut-scores and the local cut-scores highlight the potential of creating local cut-scores a method to improve diagnostic accuracy.…”