Reporting and methodological quality of COVID-19 systematic reviews needs to be improved: an evidence mapping

Abstract

Objectives To assess the reporting and methodological quality of COVID-19 systematic reviews, and to analyze trends and gaps in the quality, clinical topics, author countries, and populations of the reviews using an evidence mapping approach. Study Design and Setting A structured search for systematic reviews concerning COVID-19 was performed using PubMed, Embase, Cochrane Library, Campbell Library, Web of Science, CBM, WanFang Data, CNKI, and CQVIP from inception until June 2020. The quality of each review was assessed using the Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) checklist and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Results In total, 243 systematic reviews met the inclusion criteria, over 50% of which (128, 52.7%) were from 14 developing countries, with China contributing the most reviews (76, 31.3%). In terms of methodological quality of the studies, 30 (12.3%) were of moderate quality, 63 (25.9%) were of low quality, and 150 (61.7%) were of critically low quality. In terms of reporting quality, the median (interquartile range) PRISMA score was 14 (10–18). Regarding the topics of the reviews, 24 (9.9%) focused on the prevalence of COVID-19, 69 (28.4%) focused on the clinical manifestations, 30 (12.3%) focused on etiology, 43 (17.7%) focused on diagnosis, 65 (26.7%) focused on treatment, 104 (42.8%) focused on prognosis, and 25 (10.3%) focused on prevention. These studies mainly focused on general patients with COVID-19 (161, 66.3%), followed by children (22, 9.1%) and pregnant patients (18, 7.4%). Conclusion This study systematically evaluated the methodological and reporting quality of systematic reviews of COVID-19, summarizing and analyzing trends in their clinical topics, author countries, and study populations.

A Call For Better Methodological Quality Of Reviews On Using Artificial Intelligence For COVID-19 Detection In Medical Imaging – An Umbrella Systematic Review

“…Out of 17 available reviews published before September 1, 2020, 5 (29%) were found to be of low and the remaining 12 (71%) of critically low quality. This is also in line with Li et al [31] who assessed 63 systematic reviews (25%) to have low and 150 (62%) to have critically low quality. The authors also evaluated reporting using PRISMA [77], and the median score was 14 (10-18).…”

“…In some recent studies and clinical trials, AI has been demonstrated to match or even exceed the performance of expert radiologists, which could potentially offer expedited and less expensive diagnostics [25–30]. A recent study and meta-analysis by Li et al [31] with 31,587 identified and 82 included studies shows deep learning is capable of slightly outperforming health care professionals in detecting diseases from medical images with a pooled sensitivity of 87% (vs 86% of health care professionals) and pooled specificity of 93% (vs 91% respectively). Overlapping confidence intervals suggest that there is no statistically significant difference in performance between AI and human.…”

The rapid, massive growth of COVID-19 authors in the scientific literature

“…The quality of the published work was not assessed in our analysis, given the broad scope and huge diversity of the included papers. Nevertheless, many surveys of the quality of COVID-19 publications already exist [15,[25][26][27][28][29][30][31][32][33][34][35][36][37]. Although existing surveys of the quality of COVID-19 research do not cover all subfields of investigation and quality is often difficult to measure precisely, the consistent finding of the high prevalence of low-quality studies across very different types of study designs suggests that a large portion ( perhaps even the large majority) of the immense and rapidly growing COVID-19 literature may be of low quality.…”

The rapid, massive growth of COVID-19 authors in the scientific literature

“…Nevertheless, many surveys of the quality of COVID-19 publications already exist. [25][26][27][28][29][30][31][32][33][34][35][36][37][38] Although existing surveys of the quality of COVID-19 research do not cover all subfields of investigation and quality is often difficult to measure precisely, the consistent finding of high prevalence of low quality studies across very different types of study designs suggests that a large portion (perhaps even the large majority) of the immense and rapidly growing COVID-19 literature may be of low quality. Moreover, massive productivity has been described in the pre-COVID era, as affecting researchers across many fields 39 and may be a particular feature for COVID-19 research.…”