Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature

Abstract

On the behest of the Office of Science and Technology Policy in the White House, six institutions, including ours, have created an open research dataset called COVID-19 Research Dataset (CORD-19) to facilitate the development of question-answering systems that can assist researchers in finding relevant research on COVID-19. As of May 27, 2020, CORD-19 includes more than 100,000 open access publications from major publishers and PubMed as well as preprint articles deposited into medRxiv, bioRxiv, and arXiv. Recent years, however, have also seen question-answering and other machine learning systems exhibit harmful behaviors to humans due to biases in the training data. It is imperative and only ethical for modern scientists to be vigilant in inspecting and be prepared to mitigate the potential biases when working with any datasets. This article describes a framework to examine biases in scientific document collections like CORD-19 by comparing their properties with those derived from the citation behaviors of the entire scientific community. In total, three expanded sets are created for the analyses: 1) the enclosure set CORD-19E composed of CORD-19 articles and their references and citations, mirroring the methodology used in the renowned “A Century of Physics” analysis; 2) the full closure graph CORD-19C that recursively includes references starting with CORD-19; and 3) the inflection closure CORD-19I, that is, a much smaller subset of CORD-19C but already appropriate for statistical analysis based on the theory of the scale-free nature of the citation network. Taken together, all these expanded datasets show much smoother trends when used to analyze global COVID-19 research. The results suggest that while CORD-19 exhibits a strong tilt toward recent and topically focused articles, the knowledge being explored to attack the pandemic encompasses a much longer time span and is very interdisciplinary. A question-answering system with such expanded scope of knowledge may perform better in understanding the literature and answering related questions. However, while CORD-19 appears to have topical coverage biases compared to the expanded sets, the collaboration patterns, especially in terms of team sizes and geographical distributions, are captured very well already in CORD-19 as the raw statistics and trends agree with those from larger datasets.

A scientometric overview of CORD-19

“…In a span of a few months, it appears that most gaps in the coverage of CORD-19 have been filled. This testifies to the efforts made by the CORD-19 team to expand the coverage of the dataset [ 21 ].…”

Visibility, collaboration and impact of the Cuban scientific output on COVID-19 in Scopus

“…To retrieve the publications, Scopus ( http://www.scopus.com ) was accessed on March 12, 2021, and an advanced search was performed using a filter by country (Cuba), source (journals) and type of articles (article and review). Most of the terms used for the search were extracted from previous bibliometric articles [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] and the PubMed Medical Subject Headings (MeSH) related to the disease included in the MeSH catalog in its 2021 update were also used: COVID-19 vaccines, COVID-19 testing, COVID-19 serological testing and COVID-19 nucleic acid testing. The search strategy we used is shown in Table 1 .…”

“…Hence, in record time, an unprecedented avalanche of knowledge was generated and several publishers offered online and open access publications on the disease, which are now available to the scientific community. [4] This increase in scientific output was the subject of various bibliometric studies worldwide, [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23] in Latin America, [24][25][26][27][28] Iran, [29] India, [30] in nursing journals, [31] in relation to pediatrics [32] and a stroke, [33] as well as in comparison to other coronaviruses. [34,35] J o u r n a l P r e -p r o o f Cuban science has contributed significantly to the confrontation with COVID-19, facilitating the prediction, the improvement of therapeutic protocols and the improvement of action models for the reduction of risks and vulnerabilities in the face of the pandemic.…”

“…This increase in scientific output was the subject of various bibliometric studies worldwide [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ], in Latin America [ 24 , 25 , 26 , 27 , 28 ], Iran [ 29 ], India [ 30 ], in nursing journals [ 31 ], in relation to pediatrics [ 32 ] and a stroke [ 33 ], as well as in comparison to other coronaviruses [ 34 , 35 ].…”

The boundary-spanning mechanisms of Nobel Prize winning papers

“…On one hand, predicting significant scientific achievements is a challenging enterprise as an increasing number of studies have revealed the unpredictable nature of scientific success [13]. On the other hand, recent advances in digital libraries and scientific publication indexing services such as Dimensions, Lens.org, and Microsoft Academic Graph (in addition to the Web of Science and Scopus) provide us with an unprecedented level of digital access to hundreds of millions of scientific publications and metadata [14,15]. These resources present new opportunities for developing the next-generation computational methods for explaining and predicting breakthroughs in science [16].…”