How Am I Doing?: Evaluating Conversational Search Systems Offline

Abstract

As conversational agents like Siri and Alexa gain in popularity and use, conversation is becoming a more and more important mode of interaction for search. Conversational search shares some features with traditional search, but differs in some important respects: conversational search systems are less likely to return ranked lists of results (a SERP), more likely to involve iterated interactions, and more likely to feature longer, well-formed user queries in the form of natural language questions. Because of these differences, traditional methods for search evaluation (such as the Cranfield paradigm) do not translate easily to conversational search. In this work, we propose a framework for offline evaluation of conversational search, which includes a methodology for creating test collections with relevance judgments, an evaluation measure based on a user interaction model, and an approach to collecting user interaction data to train the model. The framework is based on the idea of “subtopics”, often used to model novelty and diversity in search and recommendation, and the user model is similar to the geometric browsing model introduced by RBP and used in ERR. As far as we know, this is the first work to combine these ideas into a comprehensive framework for offline evaluation of conversational search.

Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems

“…User simulation has been widely leveraged in the past for training the dialogue state tracking component of conversational agents using reinforcement learning algorithms, either via agenda-based or model-based simulation [19]. The highly interactive nature of conversational information access systems has also sparked renewed interest in evaluation using user simulation within the IR community [4,5,23,36,38,53]. Recently, Zhang and Balog [53] proposed a general framework for evaluating conversational recommender systems using user simulation.…”

Conversational Information Seeking

“…It is important to note that it is also possible to succeed while leaving users frustrated, as studied by Feild et al (2010). A particular end-to-end evaluation approach was recently presented by Lipani et al (2021), based on the flow of different subtopics within a conversation.…”

Evaluating Mixed-initiative Conversational Search Systems via User Simulation

“…For this reason, researchers adopt human-in-the-loop techniques to mimic human-computer interactions, and further perform human annotation to evaluate the whole system's performance (in response to human). Recent work of Lipani et al [30] propose a metric for offline evaluation of conversational search systems based on user interaction model.…”