Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic

Abstract

Background Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed clinical research from major clinical journals, comparing methodological and reporting quality of covid-19 papers published in the first wave (here defined as December 2019 to May 2020 inclusive) of the viral pandemic with non-covid papers published at the same time. Methods We reviewed research publications (print and online) from The BMJ, Journal of the American Medical Association (JAMA), The Lancet, and New England Journal of Medicine, from first publication of a covid-19 research paper (February 2020) to May 2020 inclusive. Paired reviewers were randomly allocated to extract data on methodological quality (risk of bias) and reporting quality (adherence to reporting guidance) from each paper using validated assessment tools. A random 10% of papers were assessed by a third, independent rater. Overall methodological quality for each paper was rated high, low or unclear. Reporting quality was described as percentage of total items reported. Results From 168 research papers, 165 were eligible, including 54 (33%) papers with a covid-19 focus. For methodological quality, 18 (33%) covid-19 papers and 83 (73%) non-covid papers were rated as low risk of bias, OR 6.32 (95%CI 2.85 to 14.00). The difference in quality was maintained after adjusting for publication date, results, funding, study design, journal and raters (OR 6.09 (95%CI 2.09 to 17.72)). For reporting quality, adherence to reporting guidelines was poorer for covid-19 papers, mean percentage of total items reported 72% (95%CI:66 to 77) for covid-19 papers and 84% (95%CI:81 to 87) for non-covid. Conclusions Across various measures, we have demonstrated that covid-19 research from the first wave of the pandemic was potentially of lower quality than contemporaneous non-covid research. While some differences may be an inevitable consequence of conducting research during a viral pandemic, poor reporting should not be accepted.

The rapid, massive growth of COVID-19 authors in the scientific literature

“…The quality of the published work was not assessed in our analysis, given the broad scope and huge diversity of the included papers. Nevertheless, many surveys of the quality of COVID-19 publications already exist [15,[25][26][27][28][29][30][31][32][33][34][35][36][37]. Although existing surveys of the quality of COVID-19 research do not cover all subfields of investigation and quality is often difficult to measure precisely, the consistent finding of the high prevalence of low-quality studies across very different types of study designs suggests that a large portion ( perhaps even the large majority) of the immense and rapidly growing COVID-19 literature may be of low quality.…”

The Dutch Data Warehouse, a multicenter and full-admission electronic health records database for critically ill COVID-19 patients

“…For any medical data science project, and in particular projects throughout the COVID-19 pandemic, understanding and verifying the underlying data is crucial to interpret results. Reports have expressed worries about the quality of research conducted throughout the pandemic [ 26 , 27 ]. The call for accurate, timely and reliable research data is larger than ever before.…”

The rapid, massive growth of COVID-19 authors in the scientific literature

“…Nevertheless, many surveys of the quality of COVID-19 publications already exist. [25][26][27][28][29][30][31][32][33][34][35][36][37][38] Although existing surveys of the quality of COVID-19 research do not cover all subfields of investigation and quality is often difficult to measure precisely, the consistent finding of high prevalence of low quality studies across very different types of study designs suggests that a large portion (perhaps even the large majority) of the immense and rapidly growing COVID-19 literature may be of low quality. Moreover, massive productivity has been described in the pre-COVID era, as affecting researchers across many fields 39 and may be a particular feature for COVID-19 research.…”