SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search

MacAvaney, Sean; Cohan, Arman; Goharian, Nazli

doi:10.18653/v1/2020.emnlp-main.341

Cited by 21 publications

(21 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This evidence is backed up by following scholarly publications [108,109,110,111,112,113,114,115,116,117,118,119]. I first demonstrated that neural techniques can produce effective ranking models (Hypothesis 1).…”

Section: Discussionmentioning

confidence: 98%

See 1 more Smart Citation

Effective and practical neural ranking

MacAvaney

2021

SIGIR Forum

Self Cite

View full text Add to dashboard Cite

Supervised machine learning methods that use neural networks ("deep learning") have yielded substantial improvements to a multitude of Natural Language Processing (NLP) tasks in the past decade. Improvements to Information Retrieval (IR) tasks, such as ad-hoc search, lagged behind those in similar NLP tasks, despite considerable community efforts. Although there are several contributing factors, I argue in this dissertation that early attempts were not more successful because they did not properly consider the unique characteristics of IR tasks when designing and training ranking models. I first demonstrate this by showing how large-scale datasets containing weak relevance labels can successfully replace training on in-domain collections. This technique improves the variety of queries encountered when training and helps mitigate concerns of over-fitting particular test collections. I then show that dataset statistics available in specific IR tasks can be easily incorporated into neural ranking models alongside the textual features, resulting in more effective ranking models. I also demonstrate that contextualized representations, particularly those from transformer-based language models, considerably improve neural ad-hoc ranking performance. I find that this approach is neither limited to the task of ad-hoc ranking (as demonstrated by ranking clinical reports) nor English content (as shown by training effective cross-lingual neural rankers). These efforts demonstrate that neural approaches can be effective for ranking tasks. However, I observe that these techniques are impractical due to their high query-time computational costs. To overcome this, I study approaches for offloading computational cost to index-time, substantially reducing query-time latency. These techniques make neural methods practical for ranking tasks. Finally, I take a deep dive into better understanding the linguistic biases of the methods I propose compared to contemporary and traditional approaches. The findings from this analysis highlight potential pitfalls of recent methods and provide a way to measure progress in this area going forward.

show abstract

Section: Discussionmentioning

confidence: 98%

“…Parts of Chapters 2-5 are reproductions of my jointly authored publications [108,109,110,111,112,113,114,115,116,117,118,119].…”

Section: Organizationmentioning

confidence: 99%

Effective and practical neural ranking

MacAvaney

2021

SIGIR Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…Removing documents published before 2020 (or when the pandemic began to gain widespread notice) had been previously suggested by McAvaney et al in their post-hoc analysis of their neural re-ranking system as a possible method to improve performance. 24…”

Section: Methodsmentioning

confidence: 99%

A Comparative Analysis of System Features Used in the TREC-COVID Information Retrieval Challenge

Chen

Hersh

2020

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemic has resulted in a rapidly growing quantity of scientific publications from journal articles, preprints, and other sources. The TREC-COVID Challenge was created to evaluate information retrieval methods and systems for this quickly expanding corpus. Based on the COVID-19 Open Research Dataset (CORD-19), several dozen research teams participated in over 5 rounds of the TREC-COVID Challenge. While previous work has compared IR techniques used on other test collections, there are no studies that have analyzed the methods used by participants in the TREC-COVID Challenge. We manually reviewed team run reports from Rounds 2 and 5, extracted features from the documented methodologies, and used a univariate and multivariate regression-based analysis to identify features associated with higher retrieval performance. We observed that fine-tuning datasets with relevance judgments, MS-MARCO, and CORD-19 document vectors was associated with improved performance in Round 2 but not in Round 5. Though the relatively decreased heterogeneity of runs in Round 5 may explain the lack of significance in that round, fine-tuning has been found to improve search performance in previous challenge evaluations by improving a system's ability to map relevant queries and phrases to documents. Furthermore, term expansion was associated with improvement in system performance, and the use of the narrative field in the TREC-COVID topics was associated with decreased system performance in both rounds. These findings emphasize the need for clear queries in search. While our study has some limitations in its generalizability and scope of techniques analyzed, we identified some IR techniques that may be useful in building search systems for COVID-19 using the TREC-COVID test collections.

show abstract

“…The output of Pyserini is then reranked by a T5 language model, 10 which is fine-tuned on MS MARCO, a large machine reading comprehension dataset. 18 Similarly, SLEDGE 19 uses a similar approach, but using SciBERT 13 to rerank documents.…”

Section: Background and Significancementioning

confidence: 99%

A term-based and citation network-based search system for COVID-19

et al. 2021

View full text Add to dashboard Cite

The COVID-19 pandemic resulted in an unprecedented production of scientific literature spanning several fields. To facilitate navigation of the scientific literature related to various aspects of the pandemic, we developed an exploratory search system. The system is based on automatically identified technical terms, document citations, and their visualization, accelerating identification of relevant documents. It offers a multi-view interactive search and navigation interface, bringing together unsupervised approaches of term extraction and citation analysis. We conducted a user evaluation with domain experts, including epidemiologists, biochemists, medicinal chemists, and medicine students. In general, most users were satisfied with the relevance and speed of the search results. More interestingly, participants mostly agreed on the capacity of the system to enable exploration and discovery of the search space using the graph visualization and filters. The system is updated on a weekly basis and it is publicly available at http://www.nactem.ac.uk/cord/.

show abstract

SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search

Cited by 21 publications

References 21 publications

Effective and practical neural ranking

Effective and practical neural ranking

A Comparative Analysis of System Features Used in the TREC-COVID Information Retrieval Challenge

A term-based and citation network-based search system for COVID-19

Contact Info

Product

Resources

About