Prevalence of Multiplicity and Appropriate Adjustments Among Cardiovascular Randomized Clinical Trials Published in Major Medical Journals

Khan, Muhammad Shahzeb; Khan, Maaz Shah; Ansari, Zunaira Navid; Siddiqi, Tariq Jamal; Khan, Safi U.; Riaz, Irbaz Bin; Asad, Zain Ul Abideen; Mandrola, John; Wason, James; Warraich, Haider J.; Stone, Gregg W.; Bhatt, Deepak L.; Kapadia, Samir; Kalra, Ankur

doi:10.1001/jamanetworkopen.2020.3082

Cited by 13 publications

(10 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…26 Multiplicity was found in 45.1% of the trials and of them a small number adjusted for multiple testing (20.0%). This low rate is consistent with the results of a recent analysis limited to trials in the cardiovascular field 27 and may raise questions because of the potential inflation of the type I error.…”

Section: Jama Network Open | Statistics and Research Methodssupporting

confidence: 88%

Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020

et al. 2021

Self Cite

View full text Add to dashboard Cite

IMPORTANCE Randomized clinical trials (RCTs) provide the highest level of evidence to evaluate 2 or more surgical interventions. Surgical RCTs, however, face unique challenges in design and implementation. OBJECTIVE To evaluate the design, conduct, and reporting of contemporary surgical RCTs. EVIDENCE REVIEW A literature search performed in the 2 journals with the highest impact factor in general medicine as well as 6 key surgical specialties was conducted to identify RCTs published between 2008 and 2020. All RCTs describing a surgical intervention in both experimental and control arms were included. The quality of included data was assessed by establishing an a priori protocol containing all the details to extract. Trial characteristics, fragility index, risk of bias (Cochrane Risk of Bias 2 Tool), pragmatism (Pragmatic Explanatory Continuum Indicator Summary 2 [PRECIS-2]), and reporting bias were assessed. FINDINGS A total of 388 trials were identified. Of them, 242 (62.4%) were registered; discrepancies with the published protocol were identified in 81 (33.5%). Most trials used superiority design (329 [84.8%]), and intention-to-treat as primary analysis (221 [56.9%]) and were designed to detect a large treatment effect (50.0%; interquartile range [IQR], 24.7%-63.3%). Only 123 trials (31.7%) used major clinical events as the primary outcome. Most trials (303 [78.1%]) did not control for surgeon experience; only 17 trials (4.4%) assessed the quality of the intervention. The median sample size was 122 patients (IQR, 70-245 patients). The median follow-up was 24 months (IQR, 12.0-32.0 months). Most trials (211 [54.4%]) had some concern of bias and 91 (23.5%) had high risk of bias. The mean (SD) PRECIS-2 score was 3.52 (0.65) and increased significantly over the study period. Most trials (212 [54.6%]) reported a neutral result; reporting bias was identified in 109 of 211 (51.7%). The median fragility index was 3.0 (IQR, 1.0-6.0). Multiplicity was detected in 175 trials (45.1%), and only 35 (20.0%) adjusted for multiple comparisons. CONCLUSIONS AND RELEVANCEIn this systematic review, the size of contemporary surgical trials was small and the focus was on minor clinical events. Trial registration remained suboptimal and discrepancies with the published protocol and reporting bias were frequent. Few trials controlled for surgeon experience or assessed the quality of the intervention.

show abstract

Section: Jama Network Open | Statistics and Research Methodssupporting

confidence: 88%

Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the second scenario, we considered a setting whereby the two active treatment arms consisted of the same treatment given at different doses and if either one of the treatment doses showed a statistically significant effect relative to “standard of care,” we would conclude that there was evidence of treatment effect. It would be recommended to control for FWER when interventions are related and findings are summarized into one single conclusion 46‐48 . Therefore, we performed a conservative Bonferroni adjustment 49 for the two pairwise hypotheses of the treatment effects (ie, alpha level

= 2.5 %

).…”

Section: Methods For the Simulation Studiesmentioning

confidence: 99%

Constrained randomization and statistical inference for multi‐arm parallel cluster randomized controlled trials

Zhou

Turner

Simmons

et al. 2022

Statistics in Medicine

View full text Add to dashboard Cite

A practical limitation of cluster randomized controlled trials (cRCTs) is that the number of available clusters may be small, resulting in an increased risk of baseline imbalance under simple randomization. Constrained randomization overcomes this issue by restricting the allocation to a subset of randomization schemes where sufficient overall covariate balance across comparison arms is achieved. However, for multi-arm cRCTs, several design and analysis issues pertaining to constrained randomization have not been fully investigated. Motivated by an ongoing multi-arm cRCT, we elaborate the method of constrained randomization and provide a comprehensive evaluation of the statistical properties of model-based and randomization-based tests under both simple and constrained randomization designs in multi-arm cRCTs, with varying combinations of design and analysis-based covariate adjustment strategies. In particular, as randomization-based tests have not been extensively studied in multi-arm cRCTs, we additionally develop most-powerful randomization tests under the linear mixed model framework for our comparisons. Our results indicate that under constrained randomization, both model-based and randomization-based analyses could gain power while preserving nominal type I error rate, given proper analysis-based adjustment for the baseline covariates. Randomization-based analyses, however, are more robust against violations of distributional assumptions. The choice of balance metrics and candidate set sizes and their implications on the testing of the pairwise and global hypotheses are also discussed. Finally, we caution against the design and analysis of multi-arm cRCTs with an extremely small number of clusters, due to insufficient degrees of freedom and the tendency to obtain an overly restricted randomization space.

show abstract

“…It would be recommended to control for FWER when interventions are related and findings are summarized into one single conclusion. 35,36,37 Therefore, we performed a conservative Bonferroni adjustment 4 for the two pairwise hypotheses of the treatment effects (i.e., alpha level = 2.5%).…”

Section: Null Hypotheses and Multiplicity Adjustmentmentioning

confidence: 99%

Constrained randomization and statistical inference for multi-arm parallel cluster randomized controlled trials

Zhou¹,

Turner²,

Simmons³

et al. 2021

Preprint

View full text Add to dashboard Cite

Cluster randomized controlled trials (cRCTs) are designed to evaluate interventions delivered to groups of individuals. A practical limitation of such designs is that the number of available clusters may be small, resulting in an increased risk of baseline imbalance under simple randomization. Constrained randomization overcomes this issue by restricting the allocation to a subset of randomization schemes where sufficient overall covariate balance across comparison arms is achieved with respect to a pre-specified balance metric. However, several aspects of constrained randomization for the design and analysis of multi-arm cRCTs have not been fully investigated.Motivated by an ongoing multi-arm cRCT, we provide a comprehensive evaluation of the statistical properties of model-based and randomization-based tests under both simple and constrained randomization designs in multi-arm cRCTs, with varying combinations of design and analysis-based covariate adjustment strategies. In particular, as randomization-based tests have not been extensively studied in multi-arm cRCTs, we additionally develop most-powerful permutation tests under the linear mixed model framework for our comparisons. Our results indicate that under constrained randomization, both model-based and randomization-based analyses could gain power while preserving nominal type I error rate, given proper analysis-based adjustment for the baseline covariates. The choice of balance metrics and candidateset size and their implications on the testing of the pairwise and global hypotheses are also discussed. Finally, we caution against the design and analysis of multi-arm cRCTs with an extremely small number of clusters, due to insufficient degrees of freedom and the tendency to obtain an overly restricted randomization space.

show abstract

Prevalence of Multiplicity and Appropriate Adjustments Among Cardiovascular Randomized Clinical Trials Published in Major Medical Journals

Cited by 13 publications

References 32 publications

Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020

Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020

Constrained randomization and statistical inference for multi‐arm parallel cluster randomized controlled trials

Constrained randomization and statistical inference for multi-arm parallel cluster randomized controlled trials

Contact Info

Product

Resources

About