Parameter-Efficient Sparse Retrievers and Rerankers Using Adapters

Pal, Vaishali; Lassance, Carlos; Dejean, Hervé; Clinchant, Stéphane

doi:10.1007/978-3-031-28238-6_2

Cited by 2 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, our PEFA is also applicable to both pre-trained and fine-tuned ERMs, even ones initialized from black-box LLMs. Note that PEFA is orthogonal and complement to most existing literature that aims to obtain better pre-trained or fine-tuned ERMs at the learning stage, including recent studies of the parameter-efficient fine-tuning of ERMs [28,37,44]. Finally, for the ease of discussion, we assume embeddings obtained from ERMs are unit-norm (i.e., ℓ 2 normalized), hence the inner product is equivalent to the cosine similarity.…”

Section: Problem Statementmentioning

confidence: 99%

“…To avoid the expensive full-parameter fine-tuning of ERMs for various downstream tasks, there are some preliminary studies on parameter-efficient fine-tuning of ERMs [28,37,44]. Nevertheless, as pointed out by [37], naively apply existing parameter-efficient fine-tuning methods in the NLP literature, such as Adapter [18], prefix-tuning [33] and LoRA [19], often results in limited success for ERM in the retrieval applications.…”

Section: Parameter-efficient Tuning Of Ermsmentioning

confidence: 99%

See 1 more Smart Citation

PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models

Chang,

Jiang,

Zhang

et al. 2024

Proceedings of the 17th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Embedding-based Retrieval Models (ERMs) have emerged as a promising framework for large-scale text retrieval problems due to powerful large language models. Nevertheless, fine-tuning ERMs to reach state-of-the-art results can be expensive due to the extreme scale of data as well as the complexity of multi-stages pipelines (e.g., pre-training, fine-tuning, distillation). In this work, we propose the PEFA framework, namely ParamEter-Free Adapters, for fast tuning of ERMs without any backward pass in the optimization. At index building stage, PEFA equips the ERM with a non-parametric 𝑘-nearest neighbor (kNN) component. At inference stage, PEFA performs a convex combination of two scoring functions, one from the ERM and the other from the kNN. Based on the neighborhood definition, PEFA framework induces two realizations, namely PEFA-XL (i.e., extra large) using double ANN indices and PEFA-XS (i.e., extra small) using a single ANN index. Empirically, PEFA achieves significant improvement on two retrieval applications. For document retrieval, regarding Recall@100 metric, PEFA improves not only pre-trained ERMs on Trivia-QA by an average of 13.2%, but also fine-tuned ERMs on NQ-320K by an average of 5.5%, respectively. For product search, PEFA improves the Recall@100 of the fine-tuned ERMs by an average of 5.3% and 14.5%, for PEFA-XS and PEFA-XL, respectively. Our code is available at https://github.com/ amzn/pecos/tree/mainline/examples/pefa-wsdm24.

show abstract

Section: Problem Statementmentioning

confidence: 99%