Natural language with integrated deictic and graphic gestures

Neal, Jeannette G.; Thielman, C. Y.; Dobes, Zuzana Krifka; Haller, Susan; Shapiro, Stewart

doi:10.3115/1075434.1075499

Cited by 37 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Figure 1 shows the CUBRICON system architecture [10]. CUBRICON enables a user to interact using spoken or typed natural language and gesture, displaying results using combinations of language, maps, and graphics.…”

Section: Examples Of Multimedia Information Accessmentioning

confidence: 99%

Intelligent Multimedia Communication

Maybury

Stock

Wahlster

1998

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

Abstract. Multimedia communication is a part of everyday life and its appearance in computer applications is increasing in frequency and diversity. Intelligent or knowledge based computer supported communication promises a number of benefits including increased interaction efficiency and effectiveness. This article defines the area of intelligent multimedia communication, outlines fundamental research questions, summarizes the associated scientific and technical history, identifies current challenges and concludes by predicting future breakthroughs including multilinguality. We conclude describing several new research issues that systems of systems raise. Definition of Multimedia CommunicationWe define communication as the interaction between human-human, human-system, and humaninformation. This includes interfaces to people, interfaces to applications, and interfaces to information. Following Maybury and Wahlster [1], we define:-Multimedia -physical means via which information is input, output and/or stored (e.g., interactive devices such as keyboard, mouse, displays; storage devices such as disk or CD-ROM) -Multimodal -human perceptual processes such as vision, audition, taction -Multicodal -representations used to encode atomic, elements, syntax, semantics, pragmatics and related data structures (e.g., lexicons, grammars) associated with media and modalities.The majority of computational efforts have focused on multimedia human computer interfaces. There exists a large literature and associated techniques to develop learnable, usable, transparent interfaces in general (e.g., Baecker et al. [2]). In particular, we focus here on intelligent and multimedia user interfaces (e.g.,Maybury [3]) which from the user perspective assist in tasks, are context sensitive, adapt appropriately (when, where, how) and may:

show abstract

Section: Examples Of Multimedia Information Accessmentioning

confidence: 99%

Intelligent Multimedia Communication

Maybury

Stock

Wahlster

1998

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

show abstract

“…Based on Grosz and Sidner's conversation theory [Grosz and Sidner, 1986], MIND establishes a refined discourse structure as conversation proceeds. This is different from other multimodal systems that maintain the conversation history by using a global focus space [Neal et al, 1998], segmenting a focus space based on intention [Burger and Marshall, 1993], or establishing a single dialogue stack to keep track of open discourse segments [Stent et al, 1999].…”

Section: Semantic Modelling Of Conversation Discoursementioning

confidence: 99%

Mind: A Context-Based Multimodal Interpretation Framework in Conversational Systems

Chai

Pan

Zhou

2005

Advances in Natural Multimodal Dialogue Systems

View full text Add to dashboard Cite

In a multimodal human-machine conversation, user inputs are often abbreviated or imprecise. Simply fusing multimodal inputs together may not be sufficient to derive a complete understanding of the inputs. Aiming to handle a wide variety of multimodal inputs, we are building a context-based multimodal interpretation framework called MIND (Multimodal Interpreter for Natural Dialog). MIND is unique in its use of a variety of contexts, such as domain context and conversation context, to enhance multimodal interpretation. In this chapter, we first describe a fine-grained semantic representation that captures salient information from user inputs and the overall conversation, and then present a context-based interpretation approach that enables MIND to reach a full understanding of user inputs, including those abbreviated or imprecise ones.

show abstract

“…For multimodal reference resolution, some early work keeps track of a focus space from the dialog (Grosz & Sidner, 1986) and a display model to capture all objects visible on the graphical display (Neal, Thielman, Dobes, M., & Shapiro, 1998). It then checks semantic constraints such as the type of the candidate objects being referenced and their properties for reference resolution.…”

Section: Related Workmentioning

confidence: 99%

Cognitive Principles in Robust Multimodal Interpretation

Chai¹,

Prasov²,

Qu³

2006

jair

View full text Add to dashboard Cite

Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, automated interpretation of user multimodal inputs is important. Inspired by the previous investigation on cognitive status in multimodal human machine interaction, we have developed a greedy algorithm for interpreting user referring expressions (i.e., multimodal reference resolution). This algorithm incorporates the cognitive principles of Conversational Implicature and Givenness Hierarchy and applies constraints from various sources (e.g., temporal, semantic, and contextual) to resolve references. Our empirical results have shown the advantage of this algorithm in efficiently resolving a variety of user references. Because of its simplicity and generality, this approach has the potential to improve the robustness of multimodal input interpretation.

show abstract

Natural language with integrated deictic and graphic gestures

Cited by 37 publications

References 21 publications

Intelligent Multimedia Communication

Intelligent Multimedia Communication

Mind: A Context-Based Multimodal Interpretation Framework in Conversational Systems

Cognitive Principles in Robust Multimodal Interpretation

Contact Info

Product

Resources

About