Principal investigators over-optimistically forecast scientific and operational outcomes for clinical trials

Abstract

Objective To assess the accuracy of principal investigators’ (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion. Study design and setting A short, electronic survey was used to elicit subjective probabilities within seven months of trial registration. When trial results became available, prediction skill was calculated using Brier scores (BS) and compared against uninformative prediction (i.e. predicting 50% all of the time). Results 740 PIs returned surveys (16.7% response rate). Predictions on all three events tended to exceed observed event frequency. Averaged PI skill did not surpass uninformative predictions (e.g., BS = 0.25) for primary outcomes (BS = 0.25, 95% CI 0.20, 0.30) and were significantly worse for recruitment and timeline predictions (BS 0.38, 95% CI 0.33, 0.42; BS = 0.52, 95% CI 0.50, 0.55, respectively). PIs showed poor calibration for primary outcome, recruitment, and timelines (calibration index = 0.064, 0.150 and 0.406, respectively), modest discrimination in primary outcome predictions (AUC = 0.76, 95% CI 0.65, 0.85) but minimal discrimination in the other two outcomes (AUC = 0.64, 95% CI 0.57, 0.70; and 0.55, 95% CI 0.47, 0.62, respectively). Conclusion PIs showed overconfidence in favorable outcomes and exhibited limited skill in predicting scientific or operational outcomes for their own trials. They nevertheless showed modest ability to discriminate between positive and non-positive trial outcomes. Low survey response rates may limit generalizability.

Analysis of factors leading to early termination in glioblastoma-related clinical trials

“…Problems with patient recruitment and accrual are further underscored by “Lasagna’s Law” which describes investigators’ tendency to overestimate participant goals for clinical trials [ 17 ]. In addition to overestimating predicting accrual, a recent study demonstrated that principal investigators tend to overestimate favorable trial outcomes [ 18 ]. Our results show that combined, accrual issues and trial outcome related issues (i.e., termination due to futility or safety) are responsible for 50.9% of terminations in GBM-related trials supporting previous hypotheses [ 15 , 17 , 18 ].…”

Center with or Without a Coordinator? The Coordinator as an Integral Part of a Research Team

“…Open Access Journal of Clinical Trials 2024: 16 supportive role at each stage of the trial. The supportive role manifests itself in the identification of coordinators and nurses as goalkeepers during the recruitment of older participants to the study.…”

<strong>A Maturity Model for the Scientific Review of Clinical Trial Designs and Their Informativeness </strong>

“…[21] Across a number of stakeholders working to identify design practices associated with uninformativeness, there is consensus on a core set of failures. These include principal investigators (PIs) being unrealistic or overly optimistic in their ability to set and achieve feasible and appropriate sample sizes, failure to embrace trial designs that constitute "solid science", and non-use of evidence-based disease burden and effect rates [18,22,23,24]. "Studies that failed to influence policy change or a confident next step in a go/no-go decision were associated with factors such as lack of use of common endpoints, lack of conservatism in effect estimates, not using biostatistical simulation to derive sample sizes, using unduly restrictive inclusion criteria, and avoiding use of innovative trial designs."…”