Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Abstract

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal (speech plus direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

ChatrEx: Designing Explainable Chatbot Interfaces for Enhancing Usefulness, Transparency, and Trust

“…Recently Li et. al [37] explored multi-modal strategies in the context of the existing mobile app Graphical User Interface (GUIs) for fixing Natural Language Understanding (NLU) breakdowns and command disambiguations [36]. In particular, one of their system solution (Figure 2.1.a)…”

“…We hypothesize that explaining the competencies and limitations of the chatbot using the identified intent and entity will not only aid users to recognize the breakdown but also improve transparency. Furthermore, within the breakdown decision, an indication of where the problem occurred and its possible causes would help the users more clearly understand the cause of the breakdown and repair their queries [37].…”

“…Consequently, inspired by the "research through design" [67] paradigm, we contributed in developing a minimal viable implementation of a chatbot that allowed us to explore how visual explanations should be designed. Through our implementation we were able to augment the chatbot functionality with different visual in-context explanations that could support a range of infeasible and disambiguation tasks in the user study and assess them with users, however, in the future, there is still more work that needs to be done at the intersection of ML and HCI to build upon recent work [37]. Nevertheless, our designs could be used to map user intents to specific portions of GUIs and interaction examples from other users and therefore could be adapted to other feature-rich applications besides spreadsheets that have similar UIs and menu structures.…”

“…However, complexities of natural language interactions [3, 51] and limited training sets and poor conversational understanding [2] remain to be key obstacles in fully realizing the potential of human-chatbot interaction. For example, when interacting with task-oriented chatbots, a key challenge for users is dealing with conversational dead-ends or breakdowns [5,37,36]. In fact, during a conversational breakdown, as many as 70% of users may opt to quit the task or completely abandon the chatbot, while others may try to rephrase their queries with little or no success [51].…”

Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for Smartphones

“…The combination of voice and touch enhanced the experience on the mobile devices. Besides, multimodal method were also implemented to enhance the performance on disambiguation interfaces [ 32 , 35 , 45 ].…”