Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core

Abstract

Background/Aims Pragmatic clinical trials embedded within health care systems provide an important opportunity to evaluate new interventions and treatments. Networks have recently been developed to support practical and efficient studies. Pragmatic trials will lead to improvements in how we deliver health care and promise to more rapidly translate research findings into practice. Methods The NIH Health Care Systems Collaboratory was formed to conduct pragmatic clinical trials and to cultivate collaboration across research areas and disciplines to develop best practices for future studies. Through a two-stage grant process including a pilot phase (UH2) and a main trial phase (UH3), investigators across the Collaboratory had the opportunity to work together to improve all aspects of these trials before they were launched, and to address new issues that arose during implementation. Seven Cores were created to address the various considerations, including Electronic Health Records; Phenotypes, Data Standards, and Data Quality; Biostatistics and Design Core; Patient Reported Outcomes; Health Care Systems Interactions; Regulatory/Ethics; and Stakeholder Engagement. The goal of this paper is to summarize the Biostatistics and Design Core’s lessons learned during the initial pilot phase with 7 pragmatic clinical trials conducted between 2012 and 2014. Results Methodological issues arose from the five cluster randomized trials, also called group-randomized trials, including consideration of cross-over and stepped wedge designs. We outlined general themes, challenges, and proposed solutions from the pilot phase including topics such as study design, unit of randomization, sample size, and statistical analysis. Our findings are applicable to other pragmatic clinical trials conducted within health care systems. Conclusions Pragmatic clinical trials using the UH2/UH3 funding mechanism provide an opportunity to ensure that all relevant design issues have been fully considered in order to reliably and efficiently evaluate new interventions and treatments. The integrity and generalizability of trial results can only be ensured if rigorous designs and appropriate analysis choices are an essential part of their research protocols.

Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

“…In our running example, we would be interested in testing whether the treatment effect differs between two different racial groups. Based on the linear mixed effects model (4), the scaled GLS estimator √ n(̂4 − 4 ) is asymptotically normal with mean zero and variance equal to the lower right element of Σ = lim n→∞ nΣ n , which we denote by 2 4 . This suggests the use of the z-test statistic, √ n̂4∕̂4, which will be referenced to the standard normal distribution.…”

“…x and the two ICC parameters y|x and x . Compared with the conventional CRT powered to test for the overall treatment effect, we require information of the adjusted outcome ICC, and information of X ij through two second-order parameters 2 x and x . The above expression allows us to study the relationship between sample size and distributions of X ij .…”

“…Over the past decade, CRTs are also adopted for studies embedded in the healthcare delivery systems, where there is a great interest in learning the real-world effectiveness of a system-based intervention. 2 Numerous statistical methods have been developed for the design and analysis of CRTs; see, for example, the recent methodological reviews by Turner et al 3,4 However, these previous methods have primarily focused on the overall treatment effect (OTE) averaged over the population characteristics, and little discussion has been offered to address the differential treatment effect among subpopulations. A recent systematic review by Starks et al 5 also found that only 18 out of 64 health-related CRTs published between 2010 and 2016 examined heterogeneity of treatment effect (HTE) in subgroups.…”

Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory

“…Members of the Collaboratory Core Groups (described in more detail below) worked with the PIs to facilitate the PCTs and have reported on solutions to the challenges encountered. These include a special issue on the ethics of research in usual care settings [ 17 , 18 ], a special issue on the ethical and regulatory complexities of pragmatic clinical trials [ 15 , 19 – 29 ], journal articles on initiating and implementing patient-reported outcomes measures [ 30 ], electronic health records, phenotyping, and informatics [ 31 – 33 ], stakeholder engagement and health care systems interactions [ 34 – 37 ], as well as biostatistical lessons learned on cluster and constrained randomization [ 38 , 39 ]. Members of the Collaboratory have also developed a Living Textbook on the design, conduct, and dissemination of PCTs ( http://rethinkingclinicaltrials.org/ ).…”

Addressing identification bias in the design and analysis of cluster-randomized pragmatic trials: a case study

“…Yet pragmatic trials also introduce several methodological challenges [8][9][10][11][12][13]. One major challenge in pragmatic cluster-randomized trials occurs when participants are identified after randomization, as in open-cohort or continuous recruitment designs [14][15][16].…”