Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results

Abstract

BackgroundThis work has investigated under what conditions confidence intervals around the differences in mean costs from a cluster RCT are suitable for estimation using a commonly used cluster-adjusted bootstrap in preference to methods that utilise the Huber-White robust estimator of variance. The bootstrap's main advantage is in dealing with skewed data, which often characterise patient costs. However, it is insufficiently well recognised that one method of adjusting the bootstrap to deal with clustered data is only valid in large samples. In particular, the requirement that the number of clusters randomised should be large would not be satisfied in many cluster RCTs performed to date.MethodsThe performances of confidence intervals for simple differences in mean costs utilising a robust (cluster-adjusted) standard error and from two cluster-adjusted bootstrap procedures were compared in terms of confidence interval coverage in a large number of simulations. Parameters varied included the intracluster correlation coefficient, the sample size and the distributions used to generate the data.ResultsThe bootstrap's advantage in dealing with skewed data was found to be outweighed by its poor confidence interval coverage when the number of clusters was at the level frequently found in cluster RCTs in practice. Simulations showed that confidence intervals based on robust methods of standard error estimation achieved coverage rates between 93.5% and 94.8% for a 95% nominal level whereas those for the bootstrap ranged between 86.4% and 93.8%.ConclusionIn general, 24 clusters per treatment arm is probably the minimum number for which one would even begin to consider the bootstrap in preference to traditional robust methods, for the parameter combinations investigated here. At least this number of clusters and extremely skewed data would be necessary for the bootstrap to be considered in favour of the robust method. There is a need for further investigation of more complex bootstrap procedures if economic data from cluster RCTs are to be analysed appropriately.

Propensity scores used for analysis of cluster randomized trials with selection bias: a simulation study

“…Another way to estimate the variance parameter for the treatment effect with PS methods is by bootstrapping . However in CRTs, the hierarchical structure of the data makes the choice of the bootstrapping unit difficult: at the individual level, bootstrapping implies the loss of correlation in the data , and at the cluster level, performance is good only if more than 30 clusters are randomized . Moreover, these inflated standard errors may even be higher when there is greater variability in cluster sizes.…”

Developing Appropriate Methods for Cost-Effectiveness Analysis of Cluster Randomized Trials

“…Resampling at the second stage is likely to double count the within-cluster variance because the estimated cluster means from resampling at the first stage already incorporate both within- and between-cluster variability. 23,24,34 …”

The economic evaluation of a Participatory Ergonomics programme to prevent low back and neck pain

“…The 95% CI around the mean cost differences and the uncertainty around the ICERs were estimated using bootstrapping with 5000 replications [2,9]. To graphically illustrate the uncertainty around the ICERs, we plotted each of the bootstrapped costeffect pairs in a cost-effectiveness plane.…”