Confidence intervals for single-case effect size measures based on randomization test inversion

Abstract

In the current paper, we present a method to construct nonparametric confidence intervals (CIs) for single-case effect size measures in the context of various single-case designs. We use the relationship between a two-sided statistical hypothesis test at significance level α and a 100 (1 -α) % two-sided CI to construct CIs for any effect size measure θ that contain all point null hypothesis θ values that cannot be rejected by the hypothesis test at significance level α. This method of hypothesis test inversion (HTI) can be employed using a randomization test as the statistical hypothesis test in order to construct a nonparametric CI for θ. We will refer to this procedure as randomization test inversion (RTI). We illustrate RTI in a situation in which θ is the unstandardized and the standardized difference in means between two treatments in a completely randomized single-case design. Additionally, we demonstrate how RTI can be extended to other types of single-case designs. Finally, we discuss a few challenges for RTI as well as possibilities when using the method with other effect size measures, such as rank-based nonoverlap indices. Supplementary to this paper, we provide easy-to-use R code, which allows the user to construct nonparametric CIs according to the proposed method.

Randomized single-case AB phase designs: Prospects and pitfalls

“…Statistical inference refers to assessing the statistical significance of treatment effects through hypothesis testing or by constructing confidence intervals for the parameter estimates (Heyvaert et al, 2015;Michiels, Heyvaert, Meulders, & Onghena, 2017). On the one hand, inferential procedures can be divided into parametric and nonparametric procedures, and on the other hand, they can be divided into frequentist and Bayesian procedures.…”

“…When measurement occasions are randomized to treatment conditions according to the employed randomization scheme, a statistical reference distribution for a test statistic S can be calculated. This reference distribution can be used for calculating nonparametric p values or for constructing nonparametric confidence intervals for S by inverting the RT (Michiels et al, 2017). The RT is also flexible with regard to the choice of the test statistic (Ferron & Sentovich, 2002;Onghena, 1992;Onghena & Edgington, 2005).…”

“…Note that the calculation of two-sided p values are preferable if the treatment effects can go in both directions. Alternatively, the randomization test can also be inverted, in order to obtain a nonparametric confidence interval of the observed treatment effect (Michiels et al, 2017). The benefit of calculating confidence intervals over p values is that the former conveys the same information as the latter, with the advantage of providing a range of Bplausible valuesf or the test statistic in question (du Prel, Hommel, Röhrig, & Blettner, 2009).…”

The conditional power of randomization tests for single-case effect sizes in designs with randomized treatment order: A Monte Carlo simulation study

“…Statistical inference refers to determining the statistical significance of ES measures through statistical hypothesis testing or to constructing confidence intervals for parameter estimates (Heyvaert et al, 2015;Michiels, Heyvaert, Meulders & Onghena, in press).…”

Nonparametric meta-analysis for single-case research: Confidence intervals for combined effect sizes

“…Both are equivalent in the sense that for a certain test statistic θ, the 100(1 -α)% twosided confidence interval contains all point null values of θ that cannot be rejected by a two-sided hypothesis test at significance level α. HTI can be used with either parametric hypothesis tests or nonparametric hypothesis tests such as RTs. Michiels, Heyvaert, Meulders, and Onghena (2017) already showed that nonparametric confidence intervals for single-case ESs can be constructed from the RT using HTI.…”