Empirical Synthesis of the Effect of Standard Error of Measurement on Decisions Made Within Brief Experimental Analyses of Reading Fluency

Abstract

Intervention researchers often use curriculum-based measurement of reading fluency (CBM-R) with a brief experimental analysis (BEA) to identify an effective intervention for individual students. The current study synthesized data from 22 studies that used CBM-R data within a BEA by computing the standard error of measure (SEM) for the median data point from the baseline and intervention data. The median CBM-R score from the intervention that the authors of each study identified as most effective fell within the SEM (68% confidence interval) of the baseline data approximately 30% of the time, but the ranges for the two author-identified most effective interventions overlapped over 75% of the time. Extended analyses were consistent with the BEA results for approximately three-fourths of the instances after considering the SEM of the baseline and intervention phases. Using matched passages did not improve the overlap of the ranges, but there was less overlap when the study used three data points per condition. Results emphasize the importance of considering SEM of CBM-R data when comparing interventions within a BEA. Further implications for practice and future research are included. C 2017 Wiley Periodicals, Inc.

Evaluating contextually adapted reading interventions with third-grade, Costa Rican students experiencing significant reading difficulties

“…Mean difference data have strengths and limitations for interpreting ATD data, but according to Manalov and Onghena (2017), they are the most common form of quantification in ATDs and can be particularly useful when data patterns (even nonlinear patterns) are similar across conditions. By calculating means across conditions, this also allowed for a relatively straightforward method of examining whether mean differences between conditions exceeded what might be expected simply from measurement error, which is increasingly being used in single-case experimental design (SCD) research (e.g., Burns et al, 2017;Ross & Begeny, 2015). Although there are no known standard error of measurement (SEM) data available for the specific ORF passages used in this study, tertiary analysis of mean differences relied upon known estimates of SEM for third-grade ORF passages.…”

Curriculum Based Measurement Maze: A Review

“…ORF measure is considered as the most critical measurement in R-CBM since 1982 (Deno, Mirkin, & Chiang) and therefore the existing literature on ORF, over the last 10 years, focuses particularly on issues such as its utility for reading difficulties identification in other languages (Protopappas & Skalumbakas, 2008), probe equivalence (Christ & Ardoin, 2009), the way of graphically representing progress (Christ, Silberglitt, Yeo, & Cormier, 2010), whether R-CBM measurements can be used as indicators on the impact of teaching methods used by teachers (Petscher, Cummings, Biancarosa, & Fien, 2013), how to use the data resulting from monitoring the progress for the educational decision-making process (Burns et al, 2017;Van Norman & Christ, 2016) and the degree of correlation with reading comprehension (Shin & McMaster, 2019).…”

False Discovery Rates When Engaging in Skill Versus Performance Deficit Analysis for Academic Instructional Planning

“…However, comparing so few data points—minimally two—may be insufficient in reducing decision-making errors to a level that justifies use of BEA (Andersen et al, 2013; Mercer, Harpole, Mitchell, McLemore, & Hardy, 2012). That is, BEA does not account for the uncertainty of conclusions based on the sampling distribution of the outcome (Burns et al, 2017). Importantly, this perspective on BEA assumes a true population parameter exists and error is produced from factors orthogonal to the variability produced by the independent variable (i.e., classical test theory; CTT).…”

“…Relatedly, Burns et al (2017) used existing reports to approximate the SEM of the probes used in published BEAs that focused on reading. The authors reported that approximately 25% of reviewed studies had overlapping 68% confidence intervals, highlighting the errors that may occur during BEA.…”

“…The authors reported that approximately 25% of reviewed studies had overlapping 68% confidence intervals, highlighting the errors that may occur during BEA. However, Burns et al (2017) did not consider the base rates of the examined interventions, nor did they isolate specific comparative intervention conditions (i.e., skill vs. performance), meriting further research. Furthermore, the authors recommended adjusting for possible errors by using confidence intervals.…”