QRFA: A Data-Driven Model of Information-Seeking Dialogues

Abstract

Understanding the structure of interaction processes helps us to improve information-seeking dialogue systems. Analyzing an interaction process boils down to discovering patterns in sequences of alternating utterances exchanged between a user and an agent. Process mining techniques have been successfully applied to analyze structured event logs, discovering the underlying process models or evaluating whether the observed behavior is in conformance with the known process. In this paper, we apply process mining techniques to discover patterns in conversational transcripts and extract a new model of information-seeking dialogues, QRFA, for Query, Request, Feedback, Answer. Our results are grounded in an empirical evaluation across multiple conversational datasets from different domains, which was never attempted before. We show that the QRFA model better reflects conversation flows observed in real information-seeking conversations than models proposed previously. Moreover, QRFA allows us to identify malfunctioning in dialogue system transcripts as deviations from the expected conversation flow described by the model via conformance analysis.

Towards a model for spoken conversational search

“…Different schemas have been proposed for information-seeking dialogues based on dialogue acts (DAs) [38] which try to capture the role of an utterance. In particular, schemas such as the COnversational Roles (COR) [40] and Query Request Feedback Answer (QRFA) [54] aim to provide the structure of a single dialogue contribution or move. In our study, we are interested in interactions between a user and SCS system in a more exhaustive manner: for example, utterances such as relevance feedback statements or physical actions (i.e., a mouse click to open a document).…”

“…To the best of our knowledge, only a few models include the system as an integral part of the search process [3,33,39,54]. Recently, Azzopardi et al [3] created a conceptual framework of the probable action and interaction space for conversational agents as a first step, acknowledging that their initial framework would need expansion and empirical evidence.…”

Analysis of Information-Seeking Conversations with Process Mining

“…Vakulenko et al [2] used conformance checking [3] to evaluate a new model of information-seeking behaviour. Specifically, they represented their new behavioral model using a Petri net created from labelled and structured datasets and then used standard conformance checking techniques to observe the extent to which the model adhere to the dataset under examination.…”

“…An emerging phenomenon within discussion platforms concerns the growing diffusion of chatbots. Statistics and predictions report 2 that, by 2020, 80% of enterprises will use chatbots and, by 2022, banks can automate up to 90% of their customer interaction using chatbots. Nowadays, these bots are typically used to answer simple common questions from users.…”

“…Nowadays, these bots are typically used to answer simple common questions from users. However when users request information, which require multiple rounds of conversation (usually referred to as a multi turn conversation), the chatbots are lacking structural sense 2 . Conversational process mining, has the potential to give these chatbots structural sense, and improve their capabilities.…”