Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues

Abstract

Ranking responses for a given dialogue context is a popular benchmark in which the setup is to re-rank the ground-truth response over a limited set of 𝑛 responses, where 𝑛 is typically 10. The predominance of this setup in conversation response ranking has lead to a great deal of attention to building neural re-rankers, while the first-stage retrieval step has been overlooked. Since the correct answer is always available in the candidate list of 𝑛 responses, this artificial evaluation setup assumes that there is a first-stage retrieval step which is always able to rank the correct response in its top-𝑛 list.In this paper we focus on the more realistic task of full-rank retrieval of responses, where 𝑛 can be up to millions of responses. We investigate both dialogue context and response expansion techniques for sparse retrieval, as well as zero-shot and fine-tuned dense retrieval approaches. Our findings-based on three different informationseeking dialogue datasets-reveal that a learned response expansion technique is a solid baseline for sparse retrieval. We find the best performing method overall to be dense retrieval with intermediate training-a step after the language model pre-training where sentence representations are learned-followed by fine-tuning on the target conversational data. We also investigate the intriguing phenomena that harder negatives sampling techniques lead to worse results for the fine-tuned dense retrieval models. The code and datasets are available at https://github.com/Guzpenha/transformer_ rankers/tree/full_rank_retrieval_dialogues.