Randomization Tests for Single Case Designs with Rapidly Alternating Conditions: An Analysis of p-Values from Published Experiments

Abstract

Two common barriers to applying statistical tests to single-case experiments are that single-case data often violate the assumptions of parametric tests and that random assignment is inconsistent with the logic of single-case design. However, in the case of randomization tests applied to single-case experiments with rapidly alternating conditions, neither the statistical assumptions nor the logic of the designs are violated. To examine the utility of randomization tests for single-case data, we collected a sample of published articles including alternating treatments or multielement designs with random or semi-random condition sequences. We extracted data from graphs and used randomization tests to estimate the probability of obtaining results at least as extreme as the results in the experiment by chance alone (i.e., p-value). We compared the distribution of p-values from experimental comparisons that did and did not indicate a functional relation based on visual analysis and evaluated agreement between visual and statistical analysis at several levels of α. Results showed different means, shapes, and spreads for the p-value distributions and substantial agreement between visual and statistical analysis when α = .05, with lower agreement when α was adjusted to preserve family-wise error at .05. Questions remain, however, on the appropriate application and interpretation of randomization tests for single-case designs. Keywords Visual analysis. Statistical analysis. Randomization test. Alternating treatments design. Multielement design. p-value To date, visual analysis has been the standard and accepted method for interpreting results of experimental single-case designs (Kratochwill, Levin, Horner, & Swoboda,

An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis

“…So, behavior science/analysis as a whole is not immune from threats to reproducibility related to NHST, nor do we suggest that behavior scientists/analysts reject NHST as a potentially viable scientific practice. NHST can serve a useful purpose when conducted and reported properly, particularly when focusing on measures of effect size and confidence intervals, rather than just the p-value (Greenwald et al, 1996;Hales et al, 2018;Harris, 1997;Weaver & Lloyd, 2018). In addition, Jones and Tukey (2000) outlined what they termed "a sensible approach" to significance testing, which they claimed avoids the most common problems associated with traditional NHST (see also Harris, 1997Harris, , 2016Hurlbert & Lombardi, 2009).…”

Predict, Control, and Replicate to Understand: How Statistics Can Foster the Fundamental Goals of Science

“…What is needed are statistical techniques that don't assume that all within-group differences are random deviations from a population mean. Permutation techniques (Weaver & Lloyd, 2018) and model comparison approaches offer opportunities for creative solutions, as does hierarchical Bayesian modeling. Skinner found a simpler way to minimize within-group variance in his new approach to a science of behavior.…”

Randomization tests as alternative analysis methods for behavior‐analytic data

“…In light of these practical and conceptual limitations of popular statistical methods like t and F tests when applied to behavior-analytic data, alternative analysis methods are desirable. An increasing number of researchers have come to embrace randomization statistics (e.g., Adams & Anthony, 1996;Ferron & Ware, 1994;Wampold & Furlong, 1981;Weaver & Lloyd, 2018; for review, see Huo, Heyvaert, Van den Noortgate, & Onghena, 2014), as such methods may be particularly well suited for use in behavioral research. The purpose of this article is to introduce readers to randomization methods.…”

“…Inferential statistics can help to clarify relations between variables while accounting for these sources of additional variance. Statistical methods also add to the set of tools that a researcher may use when making decisions about data when the functional relations between independent and dependent variables are not immediately clear through means of visual inspection alone (for discussion, see Weaver & Lloyd, ). Further, Huitema () cogently argued that statistical analysis of behavior‐analytic data could increase the visibility of the science to other disciplines (e.g., neuroscience, pharmacology, medicine) that place stronger emphasis on statistical outcomes when evaluating a study's results.…”

“…When single‐case research designs are used, researchers' statistical options are further limited. Because only a single subject or participant is observed, methods like t and F tests, which require samples of independent observations, are inappropriate (see Bulté & Onghena, ; Weaver & Lloyd, ). Further, autocorrelation between data points is a serious concern in single‐case research designs because of the time‐series nature of the data that they produce.…”