Information Retrieval in an Infodemic: The Case of COVID-19 Publications

Abstract

Background The COVID-19 global health crisis has led to an exponential surge in published scientific literature. In an attempt to tackle the pandemic, extremely large COVID-19–related corpora are being created, sometimes with inaccurate information, which is no longer at scale of human analyses. Objective In the context of searching for scientific evidence in the deluge of COVID-19–related literature, we present an information retrieval methodology for effective identification of relevant sources to answer biomedical queries posed using natural language. Methods Our multistage retrieval methodology combines probabilistic weighting models and reranking algorithms based on deep neural architectures to boost the ranking of relevant documents. Similarity of COVID-19 queries is compared to documents, and a series of postprocessing methods is applied to the initial ranking list to improve the match between the query and the biomedical information source and boost the position of relevant documents. Results The methodology was evaluated in the context of the TREC-COVID challenge, achieving competitive results with the top-ranking teams participating in the competition. Particularly, the combination of bag-of-words and deep neural language models significantly outperformed an Okapi Best Match 25–based baseline, retrieving on average, 83% of relevant documents in the top 20. Conclusions These results indicate that multistage retrieval supported by deep learning could enhance identification of literature for COVID-19–related questions posed using natural language.

Online health search via multi-dimensional information quality assessment based on deep language models

“…The TREC Health Misinformation Track (2019-2021) proposed further metrics to evaluate models according to the relevance, efficacy, credibility, usefulness, correctness, and supportiveness of the retrieved information [21][22][23]. These metrics are one of the first efforts to quantitatively assess the effectiveness of information retrieval engines to retrieve high-quality information, moving away from the traditional querydocument relevance paradigm [24,25].…”

Image_1.TIF

“…The COVID-19 pandemic revealed the necessity for accelerated vaccine development worldwide (3,7). In this critical context, we performed a risk analysis on the different vaccine technologies proposed by the manufacturers against COVID-19 in the early months of 2020 by using the AI-based search engine Risklick (12,13). We graded seven different risks across seven different vaccine technologies by using a Probability and Impact Matrix (as illustrated in Supplementary Figure 1, for the risks linked to safety for each vaccine).…”

“…In this context, we performed a risk analysis on the different technologies used by the pharmaceutical companies providing vaccines against COVID-19 in the early months of 2020 by using the AI-based search engine Risklick (12,13).…”

“…The Risklick AI collects and updates clinical trial data from a wide variety of sources such as the Clinical Trials Registr and datasets from the WHO each day. The metadata related to the publications are acquired from 14 international sources [BioRxiv; MedRxiv; Medline; Embase; Pubmed; Cinahl; Web of Science; Scopus; Cochrane; the International Clinical Trials Registry Platform (ICTRP); Dimensions; Living Evidence; Kaggle Cord-19 Dataset; and Google Scholar] allowing superior search performance relative to other acknowledged scientific publications search engines (12,13). The Risklick AI is able to find and process COVID-19 references more effectively in terms of precision, F1 score, and recall, compared to the baseline platforms.…”

Table_2.XLSX

“…Hence, we analyzed the risks of the different technologies to produce the vaccines proposed by the pharmaceutical companies providing vaccines against COVID-19. This analysis was performed using scientific publications from early 2020 and utilizing the artificial intelligence (AI)-based search engine Risklick (12,13). By calculating the risk score using a Probability and Impact Matrix and a semi-automated AI interface, we estimated the major risks faced by the pharmaceutical companies providing the vaccines.…”