Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Abstract

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multivector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5×, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality outof-domain collections, yielding statistically significant gains over traditional retrieval baselines.does doxycycline contain sulfa BERT tokenized (9 subword-tokens): 'does', 'do', '##xy', '##cy', '##cl', '##ine', 'contain', 'sul', '##fa'ColBERTer BOW 2 (30 saved vectors from 84 subword-tokens): photosensitivity doxycycline 12.9 sulfa 14.2 sunburned rash clothing sunlight allergic compound drugs containing 6.6 take safely wear . is no 4.7 exposed ... Fulltext: No doxycycline is not a sulfa containing compound, so you may take it safely if you are allergic to sulfa drugs. You should be aware, however, that doxycycline may cause photosensitivity, so you should wear appropriate clothing, or you may get easily sunburned or develop a rash if you are exposed to sunlight.

A Dense Representation Framework for Lexical and Semantic Matching

“…As 𝜃 increases, terms with lower weights are filtered out; see Eq. (18). Thus, the relevance score between the query and the passage decreases.…”

“…For a fair comparison, Table 6 only includes models that use the same baseline training strategy as ours. Thus, we exclude approaches that depend on other models for expansion [25,33,51], costly training techniques such as knowledge distillation [9,17,18,38,41,44], or special pretraining [11,20,34] (see Table 8 for more comparisons).…”

“…Transformer-based bi-encoders have been widely used as first-stage retrievers for text retrieval. Compared to their multi-vector counterparts [12,18,24], single-vector representation learning approaches (with a few representative techniques listed in Table 1) are promising due to their good balance between effectiveness and efficiency.…”

“…For simplicity, we do not show term weights and use the original passage (which is judged as relevant to the query) as a reference.We observe that when 𝜃 = 0, there are more than 10 matching terms between the query and the passage. As 𝜃 increases, terms with lower weights are filtered out; see Eq (18)…”

TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval

“…These campaigns follow the Cranfield paradigm [9] to create relevance judgements on the pooled output of the participating systems. Recently there has been a growing interest in evaluating the retrieval performance of retrieval models for domain-specific retrieval tasks [2,13,14,27,36] including the medical domain [22,23,35]. Domain-specific retrieval tasks often lack a reliable test collection with human relevance judgments following the Cranfield paradigm [22,27].…”