Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems

Higashinaka, Ryuichiro; Mizukami, Masahiro; Funakoshi, Kotaro; Araki, Mikiya; Tsukahara, Hiroshi; Kobayashi, Yuka

doi:10.18653/v1/d15-1268

Cited by 26 publications

(30 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These techniques have been employed to mine dialogue corpora with different purposes. For example, Higashinaka et al (2015) used clustering to group different types of comments provided by annotators and use them to anticipate dialogue breakdown. Madan and Joshi (2017) propose a clustering technique to extend the standard K‐means algorithm to simultaneously cluster user and system utterances considering their adjacency information.…”

Section: State Of the Art: User‐adapted Conversational Interfacesmentioning

confidence: 99%

Adaptive dialogue management using intent clustering and fuzzy rules

et al. 2020

View full text Add to dashboard Cite

Conversational systems have become an element of everyday life for billions of users who use speech-based interfaces to services, engage with personal digital assistants on smartphones, social media chatbots, or smart speakers. One of the most complex tasks in the development of these systems is to design the dialogue model, the logic that provided a user input selects the next answer. The dialogue model must also consider mechanisms to adapt the response of the system and the interaction style according to different groups and user profiles. Rule-based systems are difficult to adapt to phenomena that were not taken into consideration at design-time. However, many of the systems that are commercially available are based on rules, and so are the most widespread tools for the development of chatbots and speech interfaces. In this article, we present a proposal to: (a) automatically generate the dialogue rules from a dialogue corpus through the use of evolving algorithms, (b) adapt the rules according to the detected user intention. We have evaluated our proposal with several conversational systems of different application domains, from which our approach provided an efficient way for adapting a set of dialogue rules considering user utterance clusters.

show abstract

Section: State Of the Art: User‐adapted Conversational Interfacesmentioning

confidence: 99%

Adaptive dialogue management using intent clustering and fuzzy rules

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The 11 dialogs that are part of the survey were collected from examples of user and system generated responses given in Danieli and Gerbino (1995) The dialogs are a mix from early systems like ALICE (Wallace, 2009) and Eliza (Weizenbaum, 1966) with current state-of-the-art, and also a mix of task oriented and conversational dialog. We also included examples of dialog breakdown (Higashinaka et al, 2015) 1.…”

Section: Dialog Collectionmentioning

confidence: 99%

Evaluating Dialogs based on Grice’s Maxims

Jwalapuram

2017

RANLP - Student Research Workshop

View full text Add to dashboard Cite

There is no agreed upon standard for the evaluation of conversational dialog systems, which are well-known to be hard to evaluate due to the difficulty in pinning down metrics that will correspond to human judgements and the subjective nature of human judgment itself. We explored the possibility of using Grice's Maxims to evaluate effective communication in conversation. We collected some system generated dialogs from popular conversational chatbots across the spectrum and conducted a survey to see how the human judgements based on Gricean maxims correlate, and if such human judgments can be used as an effective evaluation metric for conversational dialog.

show abstract

“…Higashinaka et al proposed two taxonomies of errors in chat-oriented dialogue systems: theorydriven (Higashinaka et al, 2015a) and data-driven (Higashinaka et al, 2015b). 1 The theory-driven taxonomy is based on principles in dialogue theories that explain the cooperative behavior in human dialogues.…”

Section: Previous Taxonomies and Integrationmentioning

confidence: 99%

“…The data-driven taxonomy (Higashinaka et al, 2015b) was created by clustering comments (textual descriptions) that describe errors made by chat-oriented dialogue systems. The comments were written by researchers working on dialogue systems.…”

Section: Data-driven Taxonomymentioning

confidence: 99%

“…are chat-oriented, such taxonomies have also been proposed. Higashinaka et al (2015a;2015b) proposed two taxonomies; one is theory-driven and the other data-driven. However, the former suffers from the fact that dialogue theories for human conversation on which the taxonomy is based, such as Grice's maxims (Grice, 1975) and adjacency pairs (Schegloff and Sacks, 1973), are often not appropriate for categorizing errors made by chat-oriented dialogue systems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Taxonomy of Errors in Chat-oriented Dialogue Systems

Higashinaka

Funakoshi

Araki

et al. 2015

Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Self Cite

View full text Add to dashboard Cite

This paper presents a taxonomy of errors in chat-oriented dialogue systems. Compared to human-human conversations and task-oriented dialogues, little is known about the errors made in chat-oriented dialogue systems. Through a data collection of chat dialogues and analyses of dialogue breakdowns, we classified errors and created a taxonomy. Although the proposed taxonomy may not be complete, this paper is the first to present a taxonomy of errors in chat-oriented dialogue systems. We also highlight the difficulty in pinpointing errors in such systems.

show abstract

Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems

Cited by 26 publications

References 8 publications

Adaptive dialogue management using intent clustering and fuzzy rules

Adaptive dialogue management using intent clustering and fuzzy rules

Evaluating Dialogs based on Grice’s Maxims

Towards Taxonomy of Errors in Chat-oriented Dialogue Systems

Contact Info

Product

Resources

About