SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search

Abstract

With worldwide concerns surrounding the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), there is a rapidly growing body of scientific literature on the virus. Clinicians, researchers, and policymakers need to be able to search these articles effectively. In this work, we present a zeroshot ranking algorithm that adapts to COVIDrelated scientific literature. Our approach filters training data from another collection down to medical-related queries, uses a neural reranking model pre-trained on scientific text (SciBERT), and filters the target document collection. This approach ranks top among zeroshot methods on the TREC COVID Round 1 leaderboard, and exhibits a P@5 of 0.80 and an nDCG@10 of 0.68 when evaluated on both Round 1 and 2 judgments. Despite not relying on TREC-COVID data, our method outperforms models that do. As one of the first search methods to thoroughly evaluate COVID-19 search, we hope that this serves as a strong baseline and helps in the global crisis.

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

“…This prompted the creation of the Covid-19 Open Research Dataset (CORD-19) [81] and the TREC-COVID [60,73] benchmarking effort on one hand, and a flurry of new research and development of IR systems specializing on this task [9,69,80] on the other. In particular, MacAvaney et al [48,49] created Med-MARCO, a subset of the MS MARCO dataset that are related to medical questions. Subsequently, several groups benchmarking on TREC-COVID employed this subset for model training [46,83,88,89], while others explored finetuning on the full MS MARCO for this task [5,40,46,53,63,71].…”

“…In a meta-analysis of participating runs in the TREC-COVID challenge, Chen and Hersh [9] found the use of MS MARCO dataset for finetuning to be associated with higher retrieval performance. Similar to Med-MARCO [48,49], Hamzei et al [31] studies place-related subset of the MS MARCO dataset. Another interesting case study in this context is the application of MS MARCO to conversational search where it has been useful for both creation of new benchmarks [17,18,59] and model training [26, 39, 50, 70, 77-79, 84, 86].…”

COVID-19-Related Scientific Literature Exploration: Short Survey and Comparative Study

“…These datasets are generally constructed by a set of human curators who were provided with a list of queries (or questions) and a set of supposedly relevant documents, and the goal was to select the most pertinent documents for each query. In addition, multiple datasets have been used to train question answering models such as COVIDQA [ 11 ], COVID-19 Questions [ 12 ], COVID-QA [ 13 ], InfoBot Dataset [ 14 ], MS-MARCO [ 15 ], Med-MARCO [ 16 ], Natural Questions [ 17 ], SQuAD [ 18 ], BioASQ [ 10 ], M-CID [ 19 ] and QuAC [ 20 ]. Other datasets were used to train document summarization models.…”

“… Granularity/Levels of Representations : We also noticed that the works used different levels of granularity, which depends on the intended tasks and the available computational resources. For example, to achieve the task of document retrieval, some works opted for simple document level representations [ 53 ], while other works either used more granular representations [ 12 , 32 , 37 , 40 , 50 , 54 , 55 , 56 ] or a mix of more granular representations with document level representations [ 16 , 24 , 38 , 39 , 44 , 51 , 52 , 57 ]. Using KGs : Knowledge graphs were used in multiple works for different purposes.…”