The conditional power of randomization tests for single-case effect sizes in designs with randomized treatment order: A Monte Carlo simulation study

Abstract

The conditional power (CP) of the randomization test (RT) was investigated in a simulation study in which three different single-case effect size (ES) measures were used as the test statistic: the mean difference (MD), the Percentage of Nonoverlapping Data (PND), and the Nonoverlap of All Pairs (NAP). Furthermore, we studied the effect of the experimental design on the RT's CP for three different single-case designs with rapid treatment alternation: the completely randomized design (CRD), the randomized block design (RBD), and the restricted randomized alternation design (RRAD). As a third goal, we evaluated the CP of the RT for three types of simulated data: data generated from a standard normal distribution, data generated from a uniform distribution, and data generated from a first-order autoregressive Gaussian process. The results showed that the MD and NAP perform very similar in terms of CP whereas the PND performs substantially worse. Furthermore, the RRAD yielded marginally higher power in the RT, followed by the CRD and then the RBD. Finally, the power of the RT was almost unaffected by the type of simulated data. Based on the results of the simulation study, we recommend at least 20 measurement occasions for single-case designs with randomized treatment order that are to be evaluated with an RT using a 5% significance level. Furthermore, we do not recommend the use of the PND because of its low power in the RT.

A Simulation Study on Two Analytical Techniques for Alternating Treatments Designs

“…Therefore, the correspondence between the current findings and previous evidence is high and the conclusions about the performance of the randomization test can be generalized beyond systematic ATDs and beyond the mean difference as a test statistic. Michiels et al (2017) found that the conditional power for ATD-RR is higher than for ATD-RB and our findings are consistent for ALIV as a test statistic. Moreover, despite some notable differences between the two studies (i.e., test statistic, conditional vs. unconditional power studied, two-sided vs. one-sided alternative hypothesis), both suggest that a medium-sized effect (according to the benchmarks proposed by Harrington & Velicer, 2015) such as 2 standard deviations can be detected as statistically significant with sufficient power for ATD-RR with as few as 12 measurement occasions, whereas small effects such as 1 standard deviation require more than 30 measurement occasions.…”

“…The present study provides initial simulation evidence on the performance of ALIV+RT (Manolov & Onghena, 2017), and it provides further evidence on the performance of VSC (Lanovaz, Cardinal et al, 2017) for ATD-RR and ATD-RB. In addition, the study also extends the evidence available on the performance of randomization tests with ATDs: (a) Levin et al (2012) studied systematically alternating designs (e.g., ABABABABABAB) with 12 and 24 measurement occasions, (b) Michiels et al (2018) studied the conditional power (Keller, 2012) for ATD-RR and ATD-RB with 12 to 40 measurement occasions, and both (a) and (b) used the mean difference (not ALIV) as a test statistic. In what follows, we compare the current results with these previous findings.…”

“…Regarding the random disturbance u t term, we specified it to follow a normal distribution with a mean of zero and a standard deviation equal to 1. The simulation study on randomizations tests by Michiels, Heyvaert, and Onghena (2018) showed that the differences between independent normal error and an independent uniform error.…”

Efficacy of online Memory Specificity Training in adults with a history of depression, using a multiple baseline across participants design

“…Using randomization tests 20 randomized baselines are necessary to create the option of a minimal p value of 0.05 per participant. Regarding statistical power, with a minimum number of measures of 14 per phase, a minimum of 62 measures in total for the participant with the shortest baseline, and 20 participants, previous simulation studies indicate that this study can reach sufficient statistical power (Heyvaert et al, 2017; Michiels et al, 2018). The 20 baselines varying from 14 to 33 days were randomized and then allocated to participants who met inclusion criteria.…”

Randomized single-case AB phase designs: Prospects and pitfalls

“…The RT is also flexible with regard to the choice of the test statistic (Ferron & Sentovich, 2002;Onghena, 1992;Onghena & Edgington, 2005). For example, it is possible to use an ES measure based on standardized mean differences as the test statistic in the RT , but also ES measures based on data nonoverlap (Heyvaert & Onghena, 2014;Michiels, Heyvaert, & Onghena, 2018). This freedom to devise a test statistic that fits the research question makes the RT a versatile statistical tool for various research settings and treatment effects (e.g., with mean level differences, trends, or changes in variability; Dugard, 2014).…”