The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Abstract

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Assessment and correlates of autistic symptoms in Schizophrenia Spectrum Disorders measured with the PANSS Autism Severity Score: A systematic review

“…The global quality of included studies was high, and all studies showed a low risk of bias. This observation is in line with the recent publication of these works, which were all performed in the last 10 years: the fact that more recent studies are characterized by better methodological quality is well described in meta-analytic literature ( 77 , 78 ).…”

Efficacy of Traditional Chinese Exercise in Improving Gait and Balance in Cases of Parkinson's Disease: A Systematic Review and Meta-analysis

“…In clinical research, high-quality RCTs are the primary source of evidence for the safety and efficacy of clinical therapies. They aid in avoiding or mitigating the risk of bias in these trials (Vinkers et al, 2021). As a result, our findings imply that the significant effects of TCE on BBS and gait speed were not based on lowquality RCTs; therefore future studies must utilize high-quality RCTs to evaluate the effects of TCE on BBS and gait speed in PD.…”

The score after 10 years of registration of systematic review protocols

“…In our study, protocol registration was significantly associated with a higher impact factor of the journal publishing the SR and a more frequent self-reported use of the PRISMA guidelines, which can be an indication of higher research quality. In journals with a higher impact factor (> 10), the risk of bias was found to be consistently lower with higher levels of registration of randomized controlled trials and the use of the CONSORT statement [ 30 ]. In contrast with Ge et al (2018), we did not find an association between financial sponsorship and protocol registration.…”

“…For our secondary objective, variables included in the multivariable logistic regression to explore the association with protocol registration or publication in non-Cochrane SRs were informed by previous research within our team [ 29 , 30 ]. We assumed that protocol registration or publication was more likely in SRs with a higher number of authors and affiliations, a higher journal impact factor, a COI statement reported, and reported to using the PRISMA guidelines.…”