Flexible IR Pipelines with Capreolus

Abstract

While a number of recent open-source toolkits for training and using neural information retrieval models have greatly simplified experiments with neural reranking methods, they essentially hard code a "search-then-rerank" experimental pipeline. These pipelines consist of an efficient first-stage ranking method, like BM25, followed by a neural reranking method. Deviations from this setup often require hacks; some improvements, like adding a second reranking step that uses a more expensive neural method, are infeasible without major code changes. In order to improve the flexibility of such toolkits, we propose implementing experimental pipelines as dependency graphs of functional "IR primitives, " which we call modules, that can be used and combined as needed. For example, a neural IR pipeline may rerank results from a Searcher module that efficiently retrieves results from an Index module that it depends on. In turn, the Index depends on a Collection to index, which is provided by the pipeline. This Searcher module is self-contained: the pipeline does not need to know about or interact with the Index of the Searcher, which is transparently shared among Searcher modules when possible (e.g., a BM25 and a QL Searcher might share the same Index). Similarly, a Reranker module might depend on a Trainer (e.g., Tensorflow), feature Extractor, Tokenizer, etc. In both cases, the pipeline needs to interact only with the Reranker or Searcher directly; the complexity of their dependencies is hidden and intelligently managed. We rewrite the Capreolus toolkit to take this approach and demonstrate its use.

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

“…PyGaggle is designed specifically to work with Pyserini, but the latter was meant to be used independently, and we explicitly did not wish to "hard code" our own research agenda. This separation has made it easier for other neural IR toolkits to build on Pyserini, for example, the Caprelous toolkit [29,30].…”

DiffIR: Exploring Differences in Ranking Models' Behavior

“…Since term importance varies from model to model, DiffIR allows the user to provide weight files that indicate the weight given to specific segments of a document. Capreolus [45] can generate these weight files for several neural reranking models by following one of several strategies for producing term importance weights. OpenNIR [28] can currently generate weight files for the EPIC model [30].…”

Simplified Data Wrangling with ir_datasets

“…Recently, several tools have begun to incorporate automatic dataset acquisition. These include Capreolus [93], PyTerrier [58] and OpenNIR [55]. These reduce the user burden of finding the dataset source files and figuring out how to parse them correctly.…”

“…A document lookup API provides fast access to source documents, which is useful for recent text-based ranking models, such as those that use BERT [27]. PyTerrier [58], Capreolus [93], and OpenNIR [55] recently added support for ir_datasets, greatly expanding the number of datasets they support, and other tools like Anserini [91] can utilize our tool using the command line interface. Finally, the ir_datasets catalog 2 acts as a documentation hub, making it easy to find datasets and learn about their characteristics.…”

“…Finally, the ir_datasets catalog 2 acts as a documentation hub, making it easy to find datasets and learn about their characteristics. We intend to continue to backfill prior [93] (Cap. ), PyTerrier [58] (PT), OpenNIR [55] (ONIR), Anserini [91] (Ans.…”