Estimating the Impact of Design Standards on the Rigor of a Subset of Single-Case Research

Abstract

The implementation of research-based practices by teachers in public school classrooms is required under federal law as expressed in the Individuals With Disabilities Education Improvement Act of 2004. To aid teachers in identifying such practices, researchers conduct systematic reviews of the educational literature. Although recent attention has been given to changes in the quality of these reviews, there has been minimal discussion about changes in the quality of the studies that comprise them. Specifically, to what extent have educational policies leading to the creation of experimental design standards resulted in a change in the rigor of educational research? Using a subset of the single-case literature commonly published in special education journals, we estimate the impact of What Works Clearinghouse single-case design standards on the trend in the rigor of single-case studies using a comparative interrupted time series framework. Within this subset of single-case studies, our estimation strategy did not detect a change in the trend of the rigor of single-case research following the establishment of What Works Clearinghouse single-case design standards. Implications are discussed for practitioners and researchers. Study data, syntax, and supplemental materials are available for public use at https://osf.io/xp7wv/.

How to Be RAD: Repeated Acquisition Design Features that Enhance Internal and External Validity

“…Many researchers view the RAD as strictly a comparative design, in which they used RAD to compare the effects of two or more interventions (Kennedy, 2005;Ledford & Gast, 2018;Shepley et al, 2020). However, researchers can use a RAD for either inductive/dynamic comparisons or to address deductive/static research questions (Johnson & Cook, 2019).…”

“…At the same time, intervention researchers often rely on group designs, even when such designs are underpowered, or the novel intervention is still in the development phase. The second reason could be that a basic RAD study cannot sufficiently control for threats to internal validity like history, maturation, and testing effects, as many critiques have suggested (Kennedy, 2005;Ledford & Gast, 2018;Shepley et al, 2020). In Figure 1A, the basic RAD lacks a baseline and an ongoing control condition that would strengthen the study's internal validity.…”

The quality of case studies on new product development: state of the art and future prospects

“…Under the above guidelines, the author based on previous studies identifying broad common categories of rigor (Gibbert et al , 2008; Gibbert and Ruigrok, 2010; Goffin et al , 2019; Hoorani et al , 2019; Perks and Roberts, 2013; Runfola et al , 2017; Elsahn et al , 2020) or considering rigor and transparency (Farquhar et al , 2020; Shepley et al , 2020), along with other methods (Lincoln and Guba, 1985; Anderson, 2012; Foxall, 1988; Goffin et al , 2019; Gibbert and Ruigrok, 2010; Eisenhardt and Graebner, 2007). Some representative studies are shown in Table 1.…”

“…Rigor is an important part of scientific research, it ensures the validity of the research results and provides a solid foundation for further (Conn and Ritala, 2019; Ketokivi and Choi, 2014; Molina-Azorin and Fetters, 2020). Therefore, transparency, reliability and validity, are essential in the evaluation index system of case studies (Aguinis and Solarino, 2019; O’Kane et al , 2019; Shepley et al , 2020). Despite the acknowledged importance, there is a lack of consensus on the criteria that lead to rigorous qualitative research (Morse et al , 2016; Hoorani et al , 2019).…”

Rethinking Rigor in Multiple Baseline and Multiple Probe Designs

“…The increased attention to rigor of SCD studies is encouraging-guidance regarding what should be accepted as adequate evidence may ultimately lead to the conduct and publication of studies whose outcomes are believably attributable to the independent variables manipulated by researchers (although there is some evidence that standards have not yet resulted in subsequent research improvements; Shepley et al, 2021). One conceivable downside of the attention to rigor is the potential to disregard important pilot work, including bodies of work that consistently show the same effects but for which some aspects of rigor are difficult to accomplish.…”