TopiOCQA: Open-domain Conversational Question Answering with Topic Switching

Abstract

In a conversational question answering scenario, a questioner seeks to extract information about a topic through a series of interdependent questions and answers. As the conversation progresses, they may switch to related topics, a phenomenon commonly observed in information-seeking search sessions. However, current datasets for conversational question answering are limiting in two ways: 1) they do not contain topic switches; and 2) they assume the reference text for the conversation is given, that is, the setting is not open-domain. We introduce TopiOCQA (pronounced Tapioca), an open-domain conversational dataset with topic switches based on Wikipedia. TopiOCQA contains 3,920 conversations with information-seeking questions and free-form answers. On average, a conversation in our dataset spans 13 question-answer turns and involves four topics (documents). TopiOCQA poses a challenging test-bed for models, where efficient retrieval is required on multiple turns of the same conversation, in conjunction with constructing valid responses using conversational history. We evaluate several baselines, by combining state-of-the-art document retrieval methods with neural reader models. Our best model achieves F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of our dataset. Our dataset and code are available at https://mcgill-nlp.github.io/topiocqa.

INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions

“…Early CQA work including QuAC (Choi et al, 2018) and CoQA (Reddy et al, 2019) requires the agent to answer each user question in a conversation by reading a short passage. DoQA (Campos et al, 2020), QReCC (Anantha et al, 2021) and TopioCQA (Adlakha et al, 2022) extend the task to an open-domain setting where the knowledge source is a large document corpus. These studies only consider limited scenarios where the agent provides a direct answer based on a short text span in a single passage, or simply outputs no answer if there is no direct answer.…”

“…Recently, there are increasing interests in developing conversational information-seeking systems (Choi et al, 2018;Adlakha et al, 2022;Saeidi et al, 2018;Feng et al, 2020), where the system assists users in finding information from knowledge sources (e.g., text corpus) via multi-turn conversational interactions. One important advantage of conversational information-seeking systems is that users do not need to come up with a very descriptive query by themselves (Webb and Webber, 2009; Rieser and Lemon, 2009;Konstantinova and Orasan, 2013).…”

“…However, existing information-seeking conversation datasets rarely contain such mixed-initiative interactions. For example, most existing conversational question answering (CQA) work focuses on user-initiative interactions, where the agent simply responds to user questions with direct answers or uses no answer for out-of-scope queries (Choi et al, 2018;Reddy et al, 2019;Adlakha et al, 2022). Other work studies clarification questions using artificially-created data, failing to capture the envisioned information-seeking interactions (Saeidi et al, 2018;Feng et al, 2020;Aliannejadi et al, 2021;Guo et al, 2021).…”

“…We formulate two tasks for the conversational agent system, where it should (1) identify a set of evidence passages from Wikipedia, and (2) generate a response grounded in the evidence. Comparing with previous studies on open-domain information-seeking conversations (Anantha et al, 2021;Adlakha et al, 2022), the key challenges in our tasks come from identifying and fusing information from multiple evidence passages to construct responses that reflect various strategies. Since handling queries with multiple evidence passages or no direct answer is more open-ended, we emphasize the need for human evaluation, and propose a more systematic human evaluation protocol that considers diverse aspects including coherence, factual consistency and information comprehensiveness, with both the predicted evidence passages and response provided as the evaluation input.…”

DialDoc 2022 Shared Task: Open-Book Document-grounded Dialogue Modeling

“…Goal-oriented document-grounded dialogue systems enable end users to interactively query about domain-specific information based on the given documents. The tasks of querying document knowledge via conversational systems continue to attract a lot of attention from both research and industrial communities for various applications such as OR-ConvQA (Qu et al, 2020), MultiDoc2Dial , QReCC (Anantha et al, 2021), Topi-OCQA (Adlakha et al, 2022) and Abg-CoQA (Guo et al, 2021). The previous Shared Task (Feng, 2021) by the First DialDoc Workshop addressed the task of goal-oriented information-seeking dialogue systems in the machine reading comprehension setting, where the dialogue is aiming at querying about the information provided in a given * Work done while at IBM Research document (Feng et al, 2020).…”