Analysing Mixed Initiatives and Search Strategies during Conversational Search

Abstract

Information seeking conversations between users and Conversational Search Agents (CSAs) consist of multiple turns of interaction. While users initiate a search session, ideally a CSA should sometimes take the lead in the conversation by obtaining feedback from the user by offering query suggestions or asking for query clarifications i.e. mixed initiative. This creates the potential for more engaging conversational searches, but substantially increases the complexity of modelling and evaluating such scenarios due to the large interaction space coupled with the trade-offs between the costs and benefits of the different interactions. In this paper, we present a model for conversational search -from which we instantiate different observed conversational search strategies, where the agent elicits: (i) Feedback-First, or (ii) Feedback-After. Using 49 TREC WebTrack Topics, we performed an analysis comparing how well these different strategies combine with different mixed initiative approaches: (i) Query Suggestions vs. (ii) Query Clarifications. Our analysis reveals that there is no superior or dominant combination, instead it shows that query clarifications are better when asked first, while query suggestions are better when asked after presenting results. We also show that the best strategy and approach depends on the trade-offs between the relative costs between querying and giving feedback, the performance of the initial query, the number of assessments per query, and the total amount of gain required. While this work highlights the complexities and challenges involved in analyzing CSAs, it provides the foundations for evaluating conversational strategies and conversational search agents in batch/offline settings. CCS CONCEPTS• Information systems → Users and interactive retrieval; • Human-centered computing → HCI design and evaluation methods.

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search

“…In mixed-initiative search systems where the initiative shifts back and forth between agents and users [5,28], asking clarifying questions has received considerable attention [27,45,56,76]. Efforts have been made to investigate the role of clarifying questions in mixed-initiative systems, recognizing their potential to improve search quality and user experience [1,40,64]. To explore when to ask clarification questions, the TREC CAsT 2022 track included a task where the system can either take the initiative by posing questions or generating a response.…”

Conversational Information Seeking

“…A recent alternative to static conversational datasets is relying on simulators (Zhang and Balog, 2020;Ie et al, 2019;Aliannejadi et al, 2021a). For instance, Zhang and Balog (2020) argued that a simulator "should enable to compute an automatic assessment of the agent such that it is predictive of its performance with real users".…”

Exploring the economics of conversational search sessions

“…Conversational search systems allow the users to issue queries in natural language and can involve multiple rounds of turn-taking and clarifications to understand the users' information needs (Aliannejadi et al, 2019;Allen et al, 2001;Ghosh, 2020;Kiesel et al, 2018). Ideally, such conversations should be mixed-initiative (Aliannejadi et al, 2021;Graesser et al, 2005;Louwerse et al, 2002), where the search agent and the user take control of the conversation at different stages of the search task (Allen et al, 1999). The initiative could be the user clarifying his intent voluntarily and explaining the context of the search.…”