Miamm — A Multimodal Dialogue System Using Haptics

Reithinger, Norbert; Dirk, Fedeler,; Kumar, Ashwani; Pecourt, Elsa; Lauer, Christoph; Romary, Laurent

doi:10.1007/1-4020-3933-6_14

Cited by 19 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, as regards interactive systems that generate data visualizations more in general, the vast majority of those are not focused on natural, conversational interaction: (Gao et al, 2015) does not provide two-way communication; the number of supported query types are limited in both (Cox et al, 2001) and (Reithinger et al, 2005), while (Sun et al, 2013) uses simple NLP methods that limit the extent of natural language understanding possible. EVIZA (Setlur et al, 2016), perhaps the closest project to our own, does provide a dialogue interface for users to explore visualizations; however, EVIZA focuses on supporting a user interacting with one existing visualization, and doesn't cover creating a new visualization, modifying the existing one, or interacting with more than one visualization at a time.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Kumar¹,

Aurisano²,

Eugenio³

et al. 2017

SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue

View full text Add to dashboard Cite

The goals of our work are twofold: gain insight into how humans interact with complex data and visualizations thereof in order to make discoveries; and use our findings to develop a dialogue system for exploring data visualizations. Crucial to both goals is understanding and modeling of multimodal referential expressions, in particular those that include deictic gestures. In this paper, we discuss how context information affects the interpretation of requests and their attendant referring expressions in our data. To this end, we have annotated our multimodal dialogue corpus for context and both utterance and gesture information; we have analyzed whether a gesture co-occurs with a specific request or with the context surrounding the request; we have started addressing multimodal co-reference resolution by using Kinect to detect deictic gestures; and we have started identifying themes found in the annotated context, especially in what follows the request.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Kumar¹,

Aurisano²,

Eugenio³

et al. 2017

SEMDIAL 2017 (SaarDial) Workshop on the Semantics and Pragmatics of Dialogue

View full text Add to dashboard Cite

show abstract

“…Systems like AutoBrief (Green et al, 2004) focus on producing graphics accompanied by text; or on finding the appropriate graphics to accompany existing text (Li et al, 2013). (Cox et al, 2001;Reithinger et al, 2005) were among the first to integrate a dialogue interface into an existing information visualization system, but they support only a small range of questions. Our own Articulate maps NL queries to statistical visualizations by using very simple NLP methods.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Katagiri

Nakano

Fernández

et al. 2016

View full text Add to dashboard Cite

We extend special thanks to our Local co-Chairs, Ron Artstein and Alesia Gainer, and their team of student volunteers. We know SIGDIAL 2016 would not have been possible without Ron and Alesia, who invested so much effort in arranging the conference venue and accommodations, handling registration, making banquet arrangements, and handling numerous other preparations for the conference. The student volunteers for on-site assistance also deserve our appreciation.Ethan Selfridge, Sponsorships Chair, has earned our appreciation for recruiting and liaising with our conference sponsors, many of whom continue to contribute year after year. Sponsorships support valuable aspects of the program, such as the invited speakers and conference banquet. In recognition of this, we gratefully acknowledge the support of our sponsors: (Platinum level) Microsoft Research, Xerox and PARC, Intel, (Gold level) Facebook, (Silver level) Amazon Alexa, Interactions, Educational Testing Service, Honda Research Institute, and Yahoo!. At the same time, we thank Priscilla Rasmussen at the ACL for tirelessly handling the financial aspects of sponsorship for SIGDIAL 2016, and for securing our ISBN.iii We also thank the SIGdial board, especially officers Amanda Stent, Jason Williams and Kristiina Jokinen for their advice and support from beginning to end.Finally, we thank all the authors of the papers in this volume, and all the conference participants for making this stimulating event a valuable opportunity for growth in the research areas of discourse and dialogue. AbstractThis paper presents an end-to-end framework for task-oriented dialog systems using a variant of Deep Recurrent QNetworks (DRQN). The model is able to interface with a relational database and jointly learn policies for both language understanding and dialog strategy. Moreover, we propose a hybrid algorithm that combines the strength of reinforcement learning and supervised learning to achieve faster learning speed. We evaluated the proposed model on a 20 Question Game conversational game simulator. Results show that the proposed method outperforms the modular-based baseline and learns a distributed representation of the latent dialog state. IntroductionTask-oriented dialog systems have been an important branch of spoken dialog system (SDS) research (Raux et al., 2005; Young, 2006; Bohus and Rudnicky, 2003). The SDS agent has to achieve some predefined targets (e.g. booking a flight) through natural language interaction with the users. The typical structure of a task-oriented dialog system is outlined in Figure 1 (Young, 2006). This pipeline consists of several independently-developed modules: natural language understanding (the NLU) maps the user utterances to some semantic representation. This information is further processed by the dialog state tracker (DST), which accumulates the input of the turn along with the dialog history. The DST outputs the current dialog state and the dialog policy selects the next system action based on the dialog state. Then natural language gene...

show abstract

“…In earlier projects [27,17] we integrated different subcomponents into multimodal interaction systems. Thereby, hub-and-spoke dialogue frameworks played a major role [18].…”

Section: Related Workmentioning

confidence: 99%

A Discourse and Dialogue Infrastructure for Industrial Dissemination

Sonntag

Reithinger

Herzog

et al. 2010

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We think that modern speech dialogue systems need a prior usability analysis to identify the requirements for industrial applications. In addition, work from the area of the Semantic Web should be integrated. These requirements can then be met by multimodal semantic processing, semantic navigation, interactive semantic mediation, user adaptation/personalisation, interactive service composition, and semantic output representation which we will explain in this paper. We will also describe the discourse and dialogue infrastructure these components develop and provide two examples of disseminated industrial prototypes.

show abstract

Miamm — A Multimodal Dialogue System Using Haptics

Cited by 19 publications

References 6 publications

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue

A Discourse and Dialogue Infrastructure for Industrial Dissemination

Contact Info

Product

Resources

About