Introducing Neural Bag of Whole-Words with ColBERTer

Hofstätter, Sebastian; Khattab, Omar; Althammer, Sophia; Sertkan, Mete; Hanbury, Allan

doi:10.1145/3511808.3557367

Cited by 19 publications

(12 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• ANCE [53] and ADORE [58]: two effective dense retrieval models based on BERT-Base [13] that use the model itself to mine hard negative documents. • RocketQA [37], Margin-MSE [17], and TAS-B [18]: effective dense retrieval models that use knowledge distillation from a BERT reranking model (a cross-encoder) in addition to various techniques for negative sampling. • Contriever-FT [20]: a single vector dense retrieval model that is pre-trained for retrieval tasks and then fine-tuned on MS MARCO.…”

Section: Resultsmentioning

confidence: 99%

“…Existing single vector dense retrieval models uses a 𝑘-dimensional latent vector to represent each query or each query token [17,23,53,57]. We argue that these dense retrieval models can benefit from modeling uncertainty in representation learning.…”

Section: The Mrl Frameworkmentioning

confidence: 99%

“…Pretrained large language models (LLMs) have demonstrated promising results in various information retrieval tasks [10,17,33,57]. Therefore, we decide to adapt existing pretrained LLMs to learn a 𝑘-variate normal distribution for each given input.…”

Section: Encoder Architecturementioning

confidence: 99%

See 2 more Smart Citations

Retrieval-Enhanced Machine Learning

Zamani

Dı́az

Dehghani³

et al. 2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Dense retrieval models use bi-encoder network architectures for learning query and document representations. These representations are often in the form of a vector representation and their similarities are often computed using the dot product function. In this paper, we propose a new representation learning framework for dense retrieval. Instead of learning a vector for each query and document, our framework learns a multivariate distribution and uses negative multivariate KL divergence to compute the similarity between distributions. For simplicity and efficiency reasons, we assume that the distributions are multivariate normals and then train large language models to produce mean and variance vectors for these distributions. We provide a theoretical foundation for the proposed framework and show that it can be seamlessly integrated into the existing approximate nearest neighbor algorithms to perform retrieval efficiently. We conduct an extensive suite of experiments on a wide range of datasets, and demonstrate significant improvements compared to competitive dense retrieval models.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: The Mrl Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

Retrieval-Enhanced Machine Learning

Zamani

Dı́az

Dehghani³

et al. 2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Our baselines for comparison are the original OKAPI BM25 algorithm [24] and the BERT-based method of [54] denoted by BERT-rank. Our vector matching method is denoted by VM, and we use all varieties of text data (described in Section 2.2) and text representation vectors (described in Section 2.3).…”

Section: Methodsmentioning

confidence: 99%

Job Vacancy Ranking with Sentence Embeddings, Keywords, and Named Entities

Vanetik,

Kogan

2023

Information

View full text Add to dashboard Cite

Resume matching is the process of comparing a candidate’s curriculum vitae (CV) or resume with a job description or a set of employment requirements. The objective of this procedure is to assess the degree to which a candidate’s skills, qualifications, experience, and other relevant attributes align with the demands of the position. Some employment courses guide applicants in identifying the key requirements within a job description and tailoring their experience to highlight these aspects. Conversely, human resources (HR) specialists are trained to extract critical information from numerous submitted resumes to identify the most suitable candidate for their organization. An automated system is typically employed to compare the text of resumes with job vacancies, providing a score or ranking to indicate the level of similarity between the two. However, this process can become time-consuming when dealing with a large number of applicants and lengthy vacancy descriptions. In this paper, we present a dataset consisting of resumes of software developers extracted from a public Telegram channel dedicated to Israeli hi-tech job applications. Additionally, we propose a natural language processing (NLP)-based approach that leverages neural sentence representations, keywords, and named entities to achieve state-of-the-art performance in resume matching. We evaluate our approach using both human and automatic annotations and demonstrate its superiority over the leading resume–vacancy matching algorithm.

show abstract

“…This paper uses the loss of SPLADE with a combination that delivers the best result in our training process. 𝐿 𝑅 is the ranking loss with margin MSE for knowledge distillation [12]. [37], DeepCT [5], DeepImpact [23], and uniCOIL [10,20].…”

Section: Introductionmentioning

confidence: 99%

Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval

Qiao

Yang

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Learned sparse document representations using a transformerbased neural model has been found to be attractive in both relevance effectiveness and time efficiency. This paper describes a representation sparsification scheme based on hard and soft thresholding with an inverted index approximation for faster SPLADE-based document retrieval. It provides analytical and experimental results on the impact of this learnable hybrid thresholding scheme. CCS CONCEPTS• Information systems → Retrieval efficiency.

show abstract

Introducing Neural Bag of Whole-Words with ColBERTer

Cited by 19 publications

References 27 publications

Retrieval-Enhanced Machine Learning

Retrieval-Enhanced Machine Learning

Job Vacancy Ranking with Sentence Embeddings, Keywords, and Named Entities

Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval

Contact Info

Product

Resources

About