Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Abstract

Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s).In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.

A Tool for Conducting User Studies on Mobile Devices

“…However, there are known game elements that can be reused in many studies, for instance leader boards and points. Also, we plan to extend Omicron to enable studies on conversational search on mobile devices [4].…”

Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search

“…Different from typical ad-hoc search, here we assume that after the initial query (Q 0 ) posed by a user, a clarifying question (Q) is asked by the system, followed by a user's answer (A). We use the document corpus and single-round conversations provided by Qulac, the only large-scale conversational search dataset with clarifying questions we are aware of [1]. Qulac is built on top of the TREC Web Track 2009-2012 data and consists of 10K query-question-answer tuples for 198 TREC topics with 762 facets, with each Q&A corresponding to a facet.…”

“…We define four polarity classes: Positive P, Negative (N ), "I don't know" (idk), and Other (O). 1 Motivated by Figure 1, we use a simple yet effective heuristic to annotate positive and negative polarity in an answer. More specifically, we tag an answer with P if it contains the term "yes", and with N if it contains the term "no".…”

“…This makes the ability of a conversational search system to support mixed-initiative interactions imperative [6,8]. Such a system can assist users to refine their information need, i.e., by disclosing new information to them [8], or posing clarifying questions [1].…”

“…Answer polarity indicates whether the question points to a relevant direction or not, while answer length enables us to (noisily) identify the presence of additional information in the response. We conduct our analysis on the Qulac dataset [1], using a query likelihood model chosen because of its simplicity and transparency [7].…”