Open-Domain Question Answering Goes Conversational via Question Rewriting

Abstract

We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web pages (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.

CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning

“…We experiment with two different fixed retrievers, and show CONQRR outperforms baseline supervised models on three conversational retrieval metrics for an average of 13% performance boost on a recent open-domain CQA dataset QReCC (Anantha et al, 2021). We also observe the performance boost on three QReCC subsets from different conversation sources.…”

“…Different from our work, such datasets do not require the retrieval step. QReCC (Anantha et al, 2021) is a recently released open-domain CQA dataset of dialogues where a conversational agent first retrieves the most relevant passage(s) before generating an answer response to the a question.…”

“…Qu et al (2020) derive their conversational search dataset based on QuAC by removing all answer turns and proposes to fine-tune a dual encoder retriever for their conversational search task. In contrast, QReCC (Anantha et al, 2021) is a large-scale open-domain CQA dataset that goes beyond conversational search, with each conversation containing both user and agent utterances. The authors use a supervised QR model based on GPT2 (Radford et al, 2019) followed by a BM25 retriever for the retrieval task.…”

“…Compared to standard retrieval tasks (Nguyen et al, 2016;Kwiatkowski et al, 2019), passage retrieval for CQA poses new challenges in understanding the current user question, as each question needs to be interpreted within the dialogue context. Recently, the first large-scale open-domain CQA dataset (Anantha et al, 2021) and a related passage retrieval task have become available.…”

“…Recent work that leverages QR for conversational retrieval (Anantha et al, 2021;Dalton et al, 2020) usually collects human-rewritten queries to train a supervised QR model. However, human annotators usually rewrite conversational queries to maximize their legibility outside the dialogue context, but not necessarily to optimize the retrieval performance.…”