Evaluating Mixed-initiative Conversational Search Systems via User Simulation

Abstract

Clarifying the underlying user information need by asking clarifying questions is an important feature of modern conversational search system. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In this paper, we propose a conversational User Simulator, called USi, for automatic evaluation of such conversational search systems. Given a description of an information need, USi is capable of automatically answering clarifying questions about the topic throughout the search session. Through a set of experiments, including automated natural language generation metrics and crowdsourcing studies, we show that responses generated by USi are both inline with the underlying information need and comparable to humangenerated answers. Moreover, we make the first steps towards multiturn interactions, where conversational search systems asks multiple questions to the (simulated) user with a goal of clarifying the user need. To this end, we expand on currently available datasets for studying clarifying questions, i.e., Qulac and ClariQ, by performing a crowdsourcing-based multi-turn data acquisition. We show that our generative, GPT2-based model, is capable of providing accurate and natural answers to unseen clarifying questions in the single-turn setting and discuss capabilities of our model in the multi-turn setting. We provide the code, data, and the pre-trained model to be used for further research on the topic. 1

Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems

“…User simulation has been widely leveraged in the past for training the dialogue state tracking component of conversational agents using reinforcement learning algorithms, either via agenda-based or model-based simulation [19]. The highly interactive nature of conversational information access systems has also sparked renewed interest in evaluation using user simulation within the IR community [4,5,23,36,38,53]. Recently, Zhang and Balog [53] proposed a general framework for evaluating conversational recommender systems using user simulation.…”

“…To bridge the gap, Salle et al [36] focus on behavioral factors like cooperativeness and patience to build a more human-like simulator for information-seeking conversations. Sekulic et al [38] take this further by enabling simulated users to ask clarifying questions in a mixed-initiative setting. Sun et al [40] studied how to simulate user satisfaction in a human-like way for task-oriented tasks.…”

Minimalist and High-performance Conversational Recommendation with Uncertainty Estimation for User Preference

“…An area related to CRS is conversational search systems [74,80]. Research work in this area tend to focus on resolving ambiguity in natural language [12,59,79], on open-domain large-scale document retrieval [11,29,60,61], and often do not to utilize historic user-item interactions [28,40,67]. Conversational recommendation systems consider both users' current request and historical interactions.…”

Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents

“…Then, there are simulators based on deep neural networks, e.g., [8,10,11]. Last, we identify simulators based on large language models, e.g., [6,12,15,16]. Commonly, simulators are developed to interact with task-oriented dialogue systems such as CIA agents.…”