A Lightweight Environment for Learning Experimental IR Research Practices

Yilmaz, Zeynep Akkalyoncu; Clarke, Charles L. A.; Lin, Jimmy

doi:10.1145/3397271.3401395

Cited by 10 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The list goes on. These myriad details often trip up new researchers who are just learning systems-oriented IR evaluation methodology (motivating work such as Akkalyoncu Yilmaz et al [2]), and occasionally subtle issues confuse experienced researchers as well. 1 The explicit goal of Pyserini is to make these "easy things" easy, supporting common tasks and reducing the possibility of confusion as much as possible.…”

Section: Design Philosophymentioning

confidence: 99%

“…A number of these features for sparse retrieval are illustrated in Figure 5 and available via the IndexReader object, which can be initialized with pre-built indexes in the same way as the searcher classes. 2 In (L7-9), we illustrate how to iterate over all terms in a corpus (i.e., its dictionary) and access each term's document frequency and collection frequency. Here, we use standard Python tools to select and print out the first 10 terms alphabetically.…”

Section: Access To System Internalsmentioning

confidence: 99%

“…Anserini builds on the open-source Lucene search library and was motivated by the desire to better align academic research with the practice of building real-world search applications; see, for example, Grand et al [9]. More recently, we recognized that Anserini's reliance on the Java Virtual Machine (due to Lucene), greatly limited its reach [2,3], as Python has emerged as the language of choice for both data scientists and researchers. This is particularly the case for work on deep learning today, since the major toolkits (PyTorch [22] and Tensorflow [1]) have both adopted Python as their front-end language.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

Lin¹,

Ma²,

Lin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Pyserini is an easy-to-use Python toolkit that supports replicable IR research by providing effective first-stage retrieval in a multistage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. We also describe how our group has built a culture of replicability through shared norms and tools that enable rigorous automated testing.

show abstract

Section: Design Philosophymentioning

confidence: 99%

Section: Access To System Internalsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

Lin¹,

Ma²,

Lin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…More broadly, with its diverse and mature ecosystem, Python has emerged as the language of choice for most data scientists today. Anticipating this gap, we have been working on Pyserini, 2 Python bindings for Anserini, since late 2019 (Yilmaz et al, 2020). Pyserini is released as a well-documented, easy-to-use Python module distributed via PyPI and easily installable via pip.…”

Section: Abstractearchmentioning

confidence: 99%

Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset

Zhang¹,

Gupta²,

Tang³

et al. 2020

Proceedings of the First Workshop on Scholarly Document Processing

Self Cite

View full text Add to dashboard Cite

We present Covidex, a search engine that exploits the latest neural ranking models to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI. Our system has been online and serving users since late March 2020. The Covidex is the user application component of our three-pronged strategy to develop technologies for helping domain experts tackle the ongoing global pandemic. In addition, we provide robust and easy-to-use keyword search infrastructure that exploits mature fusion-based methods as well as standalone neural ranking models that can be incorporated into other applications. These techniques have been evaluated in the multi-round TREC-COVID challenge: Our infrastructure and baselines have been adopted by many participants, including some of the best systems. In round 3, we submitted the highest-scoring run that took advantage of previous training data and the secondhighest fully automatic run. In rounds 4 and 5, we submitted the highest-scoring fully automatic runs.

show abstract

“…However, Python is the main language for PyTorch (Paszke et al, 2019) and TensorFlow (Abadi et al, 2016), the two most popular neural network toolkits today, and more broadly, Python has emerged as the language of choice for applied machine learning today in part due to its diverse and mature ecosystem. Pyserini (Yilmaz et al, 2020) 2 bridges the gap between the JVM and Python by providing a Python interface to Anserini. Together, Anserini and Pyserini provide basic keyword search capabilities to arbitrary corpora, which include tools to fetch raw document texts as well as utilities to access various term statistics.…”

Section: The Covidex Stackmentioning

confidence: 99%

Cydex: Neural Search Infrastructure for the Scholarly Literature

Ding¹,

Zhang²,

Lin³

2020

Proceedings of the First Workshop on Scholarly Document Processing

Self Cite

View full text Add to dashboard Cite

Cydex is a platform that provides neural search infrastructure for domain-specific scholarly literature. The platform represents an abstraction of Covidex, our recently developed full-stack open-source search engine for the COVID-19 Open Research Dataset (CORD-19) from AI2. While Covidex takes advantage of the latest best practices for keyword search using the popular Lucene search library as well as stateof-the-art neural ranking models using T5, parts of the system were hard coded to only work with CORD-19. This paper describes our efforts to generalize Covidex into Cydex, which can be applied to scholarly literature in different domains. By decoupling corpusspecific configurations from the frontend implementation, we are able to demonstrate the generality of Cydex on two very different corpora: the ACL Anthology and a collection of hydrology abstracts. Our platform is entirely open source and available at cydex.ai.

show abstract

A Lightweight Environment for Learning Experimental IR Research Practices

Cited by 10 publications

References 7 publications

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset

Cydex: Neural Search Infrastructure for the Scholarly Literature

Contact Info

Product

Resources

About