Identifying and managing problematic trials: A research integrity assessment tool for randomized controlled trials in evidence synthesis

Abstract

Evidence synthesis findings depend on the assumption that the included studies follow good clinical practice and results are not fabricated or false.Studies which are problematic due to scientific misconduct, poor research practice, or honest error may distort evidence synthesis findings. Authors of evidence synthesis need transparent mechanisms to identify and manage problematic studies to avoid misleading findings. As evidence synthesis authors of the Cochrane COVID-19 review on ivermectin, we identified many problematic studies in terms of research integrity and regulatory compliance. Through iterative discussion, we developed a research integrity assessment (RIA) tool for randomized controlled trials for the update of this Cochrane review. In this paper, we explain the rationale and application of the RIA tool in this case study. RIA assesses six study criteria: study retraction, prospective trial registration, adequate ethics approval, author group, plausibility of methods (e.g., randomization), and plausibility of study results. RIA was used in the Cochrane review as part of the eligibility check during screening of potentially eligible studies. Problematic studies were excluded and studies with open questions were held in awaiting classification until clarified. RIA decisions were made independently by two authors and reported transparently. Using the RIA tool resulted in the exclusion of >40% of studies in the first update of the review. RIA is a complementary tool prior to assessing "Risk of Bias" aiming to establish the integrity and authenticity of studies. RIA provides a platform for urgent development of a standard approach to identifying and managing problematic studies.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

“…However, because the veracity of included RCTs is not usually considered, systematic reviews may unintentionally act as a pipeline for false data with the risk that this will influence care. While the need to prevent problematic studies from contributing to systematic reviews is recognised, with several recent laudable efforts to tackle the issue (18)(19)(20), there is currently limited agreement on how this should be done. The INSPECT-SR project will develop a tool for evaluating the trustworthiness studies, backed by empirical evidence and expert consensus.…”

Improving trustworthiness in research in women's health: A collective effort by OBGYN Editors

“…There have been several separate strategies proposed to prevent the publication of flawed RCTs and systematic reviews. [5][6][7][8][9][10][11][12][13][14][15][16] We have agreed to implement a bundle of strategies 17 which will be required upon submission to our journals (Table 1). Nonadherence to these criteria will result in rejection.…”

Improving trustworthiness in research in women's health: A collective effort by OBGYN Editors

“…There have been several separate strategies proposed to prevent the publication of flawed RCTs and systematic reviews. [5][6][7][8][9][10][11][12][13][14][15][16] We have agreed to implement a bundle of strategies 17 which will be required upon submission to our journals (Table 1). Non-adherence to these criteria will result in rejection.…”