BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Abstract

Neural IR models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their generalization capabilities. To address this, and to allow researchers to more broadly establish the effectiveness of their models, we introduce BEIR (Benchmarking IR), a heterogeneous benchmark for information retrieval. We leverage a careful selection of 17 datasets for evaluation spanning diverse retrieval tasks including open-domain datasets as well as narrow expert domains. We study the effectiveness of nine state-of-the-art retrieval models in a zero-shot evaluation setup on BEIR, finding that performing well consistently across all datasets is challenging.Our results show BM25 is a robust baseline and Reranking-based models overall achieve the best zero-shot performances, however, at high computational costs. In contrast, Denseretrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. In this work, we extensively analyze different retrieval models and provide several suggestions that we believe may be useful for future work. BEIR datasets and code are available at https://github.com/UKPLab/beir.

A Proposed Conceptual Framework for a Representational Approach to Information Retrieval

“…In fact, we already know the answer, at least in part: learned representations often perform terribly in out-of-distribution settings when applied in a zero-shot manner. Evidence comes from the BEIR benchmark [Thakur et al, 2021], which aims to evaluate the effectiveness of dense retrieval models across diverse domains. Results show that, in many cases, directly applying a dense retrieval model trained on one dataset to another dataset sometimes yields effectiveness that is worse than BM25.…”

“…For example, Li et al [2021] proposed model uncertainty fusion as a solution. The BEIR benchmark [Thakur et al, 2021] provides a resource to evaluate progress, and the latest results show that learned sparse representations are able to outperform BM25 [Formal et al, 2021a]. At a high level, there are at least three intertwined research questions:…”

Unsupervised Dense Information Retrieval with Contrastive Learning

“…A potential solution is to train a dense retriever on a large retrieval dataset such as MS-MARCO, and then apply it to new domains, a setting refered to as zero-shot. Unfortunately, in this setting dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al, 2021).…”

Establishing Strong Baselines for TripClick Health Retrieval

“…Here, neural approaches lead to enormous effectiveness gains over traditional techniques [8,13,20,24]. A valid concern is the generalizability and applicability of the developed techniques to other domains and settings [16,14,31,34].…”