A review of cluster randomized trials found statistical evidence of selection bias

Bolzern, Jaime; Mnyama, Nandi; Bosanquet, Kate; Torgerson, David

doi:10.1016/j.jclinepi.2018.03.010

Cited by 29 publications

(36 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The recommendation to identify and recruit participants before randomisation, if possible, has not been implemented in many trials. Indeed, a recent review of 23 cluster RCTs published between 2015 and 2017 found only four (17%) recruited participants before randomisation [10]. Whilst the recommendation of recruitment before randomisation, if possible, should remain, perhaps CONSORT guidance for cluster trials should include the suggestion of baseline testing of patient level data, as it is currently silent on this issue.…”

Section: Resultsmentioning

confidence: 99%

“…In the following we will examine two case studies: one a cluster randomised trial and the other an individually randomised trial. Both of these were identified from a recent review by Bolzern [10] and colleagues. The aim of these case studies is to demonstrate the distribution of baseline p values in a study where there is no evidence of selection bias in an individually randomised trial as recruitment has occurred before randomisation compared with a trial where recruitment has taken place after randomisation, which is often the case in a cluster RCT.…”

Section: Mainmentioning

confidence: 99%

See 1 more Smart Citation

Baseline testing in cluster randomised controlled trials: should this be done?

Bolzern

Mitchell

Torgerson

2019

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

Background Comparisons of baseline covariates in randomised controlled trials whilst often undertaken is regarded by many as an exercise in futility. Because of randomisation the null hypothesis is true for baseline comparisons and therefore any differences will occur by chance. However, this is only the case if allocations are not known in advance of recruitment. If this occurs then selection bias at randomisation may be present and it is possible that the statistical testing of covariates may unveil selection bias. In this paper we show that this is particularly the case for cluster randomised trials when post-randomised recruitment often occurs and can lead to selection bias. Main text We take a recently published cluster randomised trial that has suffered from selection bias due to differential recruitment and calculate baseline p values. We show that statistically significant imbalances of p < 0.0001 occurred in 5 of the 10 covariates. In comparison for an individually randomised trial that had no evidence of selection bias only 1 p value of p < 0.05 out of 20 tests was observed. Had baseline p values for the cluster trial been presented to journal editors, reviewers and readers then the results of the trial might have been treated with more caution. Conclusion We argue that the blanket ban of baseline testing as advocated by some may reduce the chance of identifying deficient cluster randomised trials and this opposition should be reconsidered for cluster trials.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Mainmentioning

confidence: 99%

Baseline testing in cluster randomised controlled trials: should this be done?

Bolzern

Mitchell

Torgerson

2019

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

show abstract

“…l As for many cluster RCTs, there is a risk of selection bias, with potentially different cohorts of patients recruited in different groups. 116 In the cluster RCT, there was an indication that refusals of consent were higher in the intervention sites during the intervention period than during other periods. We attempted to mitigate this issue by providing the option for patients to consent to receive solely the follow-up questionnaire, but this was rarely taken up.…”

Section: Limitationsmentioning

confidence: 99%

A nurse-led, preventive, psychological intervention to reduce PTSD symptom severity in critically ill patients: the POPPI feasibility study and cluster RCT

Mouncey

Wade

Richards-Belle

et al. 2019

Health Serv Deliv Res

View full text Add to dashboard Cite

Background High numbers of patients experience severe acute stress in critical care units. Acute stress has been linked to post-critical care psychological morbidity, including post-traumatic stress disorder (PTSD). Previously, a preventive, complex psychological intervention [Psychological Outcomes following a nurse-led Preventative Psychological Intervention for critically ill patients (POPPI)] was developed by this research team, to be led by nurses, to reduce the development of PTSD symptom severity at 6 months. Objectives The objectives were to (1) standardise and refine the POPPI intervention, and, if feasible, (2) evaluate it in a cluster randomised clinical trial (RCT). Design Two designs were used – (1) two feasibility studies to test the delivery and acceptability (to patients and staff) of the intervention, education package and support tools, and to test the trial procedures (i.e. recruitment and retention), and (2) a multicentre, parallel-group, cluster RCT with a baseline period and staggered roll-out of the intervention. Setting This study was set in NHS adult, general critical care units. Participants The participants were adult patients who were > 48 hours in a critical care unit, receiving level 3 care and able to consent. Interventions The intervention comprised three elements – (1) creating a therapeutic environment in critical care, (2) three stress support sessions for patients identified as acutely stressed and (3) a relaxation and recovery programme for patients identified as acutely stressed. Main outcome measures Primary outcome – patient-reported symptom severity using the PTSD Symptom Scale – Self Report (PSS-SR) questionnaire (to measure clinical effectiveness) and incremental costs, quality-adjusted life-years (QALYs) and net monetary benefit at 6 months (to measure cost-effectiveness). Secondary outcomes – days alive and free from sedation to day 30; duration of critical care unit stay; PSS-SR score of > 18 points; depression, anxiety and health-related quality of life at 6 months; and lifetime cost-effectiveness. Results (1) A total of 127 participants were recruited to the intervention feasibility study from two sites and 86 were recruited to the RCT procedures feasibility study from another two sites. The education package, support tools and intervention were refined. (2) A total of 24 sites were randomised to the intervention or control arms. A total of 1458 participants were recruited. Twelve sites delivered the intervention during the intervention period: > 80% of patients received two or more stress support sessions and all 12 sites achieved the target of > 80% of clinical staff completing the POPPI online training. There was, however, variation in delivery across sites. There was little difference between baseline and intervention periods in the development of PTSD symptom severity (measured by mean PSS-SR score) at 6 months for surviving patients in either the intervention or the control group: treatment effect estimate −0.03, 95% confidence interval (CI) −2.58 to 2.52; p = 0.98. On average, the intervention decreased costs and slightly improved QALYs, leading to a positive incremental net benefit at 6 months (£835, 95% CI −£4322 to £5992), but with considerable statistical uncertainty surrounding these results. There were no significant differences between the groups in any of the secondary outcomes or in the prespecified subgroup analyses. Limitations There was a risk of bias because different consent processes were used and as a result of the lack of blinding, which was mitigated as far as possible within the study design. The intervention started later than anticipated. Patients were not routinely monitored for delirium. Conclusions Among level 3 patients who stayed > 48 hours in critical care, the delivery of a preventive, complex psychological intervention, led by nurses, did not reduce the development of PTSD symptom severity at 6 months, when compared with usual care. Future work Prior to development and evaluation of subsequent psychological interventions, there is much to learn from post hoc analyses of the cluster RCT rich quantitative and qualitative data. Trial registration This trial is registered as ISRCTN61088114 and ISRCTN53448131. Funding This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 23, No. 30. See the NIHR Journals Library website for further project information.

show abstract

“…In parallel-CRTs, empirical evidence shows that this can lead to differential recruitment between study arms. 22 , 23 To our knowledge, no studies have empirically examined risks of identification and recruitment bias in SW-CRTs. However, the implications of these biases in SW-CRTs are more difficult to assess than in parallel-CRTs, because of the potential influence of not only the knowledge of the intervention at the time of recruitment, but the influence of the knowledge of when the transition to the intervention will occur.…”

Section: Risks Of Bias In An Sw-crtmentioning

confidence: 99%

Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice?

Hemming

Taljaard

2020

International Journal of Epidemiology

View full text Add to dashboard Cite

The stepped-wedge cluster randomized trial (SW-CRT) involves the sequential transition of clusters (such as hospitals, public health units or communities) from control to intervention conditions in a randomized order. The use of the SW-CRT is growing rapidly. Yet the SW-CRT is at greater risks of bias compared with the conventional parallel cluster randomized trial (parallel-CRT). For this reason, the CONSORT extension for SW-CRTs requires that investigators provide a clear justification for the choice of study design. In this paper, we argue that all other things being equal, the SW-CRT is at greater risk of bias due to misspecification of the secular trends at the analysis stage. This is particularly problematic for studies randomizing a small number of heterogeneous clusters. We outline the potential conditions under which an SW-CRT might be an appropriate choice. Potentially appropriate and often overlapping justifications for conducting an SW-CRT include: (i) the SW-CRT provides a means to conduct a randomized evaluation which otherwise would not be possible; (ii) the SW-CRT facilitates cluster recruitment as it enhances the acceptability of a randomized evaluation either to cluster gatekeepers or other stakeholders; (iii) the SW-CRT is the only feasible design due to pragmatic and logistical constraints (for example the roll-out of a scare resource); and (iv) the SW-CRT has increased statistical power over other study designs (which will include situations with a limited number of clusters). As the number of arguments in favour of an SW-CRT increases, the likelihood that the benefits of using the SW-CRT, as opposed to a parallel-CRT, outweigh its risks also increases. We argue that the mere popularity and novelty of the SW-CRT should not be a factor in its adoption. In situations when a conventional parallel-CRT is feasible, it is likely to be the preferred design.

show abstract

A review of cluster randomized trials found statistical evidence of selection bias

Cited by 29 publications

References 8 publications

Baseline testing in cluster randomised controlled trials: should this be done?

Baseline testing in cluster randomised controlled trials: should this be done?

A nurse-led, preventive, psychological intervention to reduce PTSD symptom severity in critically ill patients: the POPPI feasibility study and cluster RCT

Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice?

Contact Info

Product

Resources

About