Boosted Dense Retriever

Abstract

We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current ensemble. The final representation is the concatenation of the output vectors of all the component models, making it a drop-in replacement for standard dense retrievers at test time. Dr-Boost enjoys several advantages compared to standard dense retrieval models. It produces representations which are 4x more compact, while delivering comparable retrieval results. It also performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. In practice, this can make the difference between serving indices from disk versus from memory, paving the way for much cheaper deployments. 1

Intelligent knowledge base search tool using large language model and graph neural network

“…Currently, each component model in our architecture naively samples a subset of negative edges, whereas DrBoost could intelligently adjust the sampling distributions, potentially leading to improved results. 8 Finally, we plan to further enhance the model by incorporating user feedback through an application hosting this model. User interactions provide multiple avenues for model improvement.…”

“…Recent works have been designed to tackle the Question-answering task and require specific q uestions, a nswers, a nd p assages t o b e organized within the dataset. 8,14 While our document retrieval task is related, the concern of sensitive organizational data prevents us from using automatic labeling techniques and in general, leveraging the learned priors from internetscale models. 15,16 The alternative is a labor-intensive data annotation effort e ngaging m any s ubject matter experts.…”

“…Ensemble models have been shown to improve performance and efficiency in space and time, allowing for compact representations and faster training/inference. 7,8 These models employ a boosting mechanism, a strategy pioneered by Freund & Schapire. 9 They showed that if the weak hypothesis of each model in the ensemble is at least better than random guessing, then the combined hypothesis of the ensemble improves at an exponential rate.…”