Comparison of Overlap Methods for Quantitatively Synthesizing Single-Subject Data

Abstract

Four overlap methods for quantitatively synthesizing single-subject data were compared to visual analysts’ judgments. The overlap methods were percentage of nonoverlapping data, pairwise data overlap squared, percentage of data exceeding the median, and percentage of data exceeding a median trend. Visual analysts made judgments about 160 A-B data sets selected randomly from the Journal of Applied Behavior Analysis . The four overlap methods were compared for data sets in which all visual analysts agreed a change in data occurred or a change did not occur across conditions. Each overlap method had unacceptably high levels of errors. Given the findings and weaknesses of the overlap methods, their use should be abandoned. The desirable characteristics of a quantitative synthesis method are described.

Systematic Review of Functional Analysis and Treatment of Elopement (2000–2015)

“…Effect Sizes There is considerable debate regarding the validity of measures of effect size (e.g., standardized mean difference [SMD], regression-based techniques, overlap methods) with single-subject data (Olive & Franco, 2008;Olive & Smith, 2005;Parker, Vannest & Davis, 2011;Rakap, 2015;Wolery, Busick, Reichow, & Barton, 2010). However, we chose SMD over proposed alternatives (e.g., overlap methods) for several reasons.…”

“…First, there remains precedent for calculating SMD with single-subject data (e.g., Beeson & Robey, 2006;Boyle, Samaha, Slocum, Hoffmann, & Bloom, 2016;Gierut, Morrissette, & Dickinson, 2015), and some authors continue to recommend SMD despite issues with "serial dependency" (e.g., Olive & Smith, 2005). Second, many of the overlap methods fail to assess relative magnitude of treatment effects (Wolery et al, 2010) when those effects exceed a certain minimum (i.e., a ceiling effect). In other words, two treatments may produce the same statistic even though one intervention was more effective than the other.…”

“…Cohen (1988) suggested that d statistics of 0.2, 0.5., and 0.8 represent small, medium, and large effects, respectively. However, these benchmarks are not appropriate for data collected within single-subject designs (Gieret & Morrisette, 2011;Wolery et al, 2010), as SMD tend to be inflated with single-subject data. Instead, we partitioned effect sizes in the current review into three groups after first excluding major outliers (which resulted in the exclusion of three data points).…”

Weighting strategies in the meta-analysis of single-case studies

“…In the review of single-case meta-analyses performed by Beretvas and Chung (2008b) (Scruggs & Mastropieri, 2013), apart from the agreement with visual analysis in absence of effect (Wolery, Busick, Reichow, & Barton, 2010). The reasons for preferring the NAP are 1) it does not depend on a single extreme baseline measure; 2) in simulation studies, the NAP has also been shown to perform well in presence of autocorrelation (Manolov, Solanas, Sierra, & Evans, 2011), in contrast with the PND (Manolov, Solanas, & Leiva, 2010); 3) the NAP and the PND show similar distributions of typical values according to the review by Parker, Vannest, and Davis (2011) using real behavioral data; and 4) the critical reason for selecting the NAP was the fact that the PND does not have a known sampling distribution , which makes impossible using the most widely accepted weight for group-design studies; in contrast, there is an expression for the variance of the NAP as shown below.…”

Choosing Among Techniques for Quantifying Single-Case Intervention Effectiveness

“…For example, PNCD was selected instead of the Percent of Nonoverlapping Data (Scruggs et al, 1987) as a previous study had shown PNCD to be superior to PND (Manolov & Solanas, 2009). Another related method, the Percentage of data points Exceeding the Median (Ma, 2006) was excluded due to poor results Wolery, Busick, Reichow, & Barton, 2010).…”