Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

Oviatt, Sharon; Cohen, Phil; Wu, Lizhong; Duncan, Lisbeth; Suhm, Bernhard; Bers, Josh; Holzman, Thomas G.; Winograd, Terry; Landay, James A.; Larson, Jim; Ferro, David

doi:10.1207/s15327051hci1504_1

Cited by 275 publications

(150 citation statements)

References 72 publications

Supporting

Mentioning

139

Contrasting

Unclassified

Order By: Relevance

“…Tyto senzory jsou například: interakce na bázi pera (S.L. Oviatt, 2000), počítačová myš a klávesnice, joystick, senzory pro sledování pohybu, hmatové, tlakové či čichové a chuťové senzory (Fakhreddine Karray, 2008).…”

Section: Modalita Hci Interakceunclassified

The Interactivity of Ict Tools

Kortan¹,

Prokýšek²

2015

JTIE

View full text Add to dashboard Cite

Section: Modalita Hci Interakceunclassified

The Interactivity of Ict Tools

Kortan¹,

Prokýšek²

2015

JTIE

View full text Add to dashboard Cite

“…For example, a large amount of training data is required to build this type of systems, and multimodal training corpora are not readily available [39].…”

Section: Error Reduction By Designmentioning

confidence: 99%

“…The other fundamental type of multimodal architecture is called "semantic level architecture", and is more appropriate for modalities less tightly coupled such as speech and pen [39]. In this type of architecture, information is typically integrated at the semantic or pragmatic level [50].…”

Section: Automatic Detectionmentioning

confidence: 99%

Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

Bourguet

2006

Signal Processing

View full text Add to dashboard Cite

In this paper, we survey the different types of error-handling strategies that have been described in the literature on recognition-based human-computer interfaces. A wide range of strategies can be found in spoken human-machine dialogues, handwriting systems, and multimodal natural interfaces.We then propose a taxonomy for classifying error-handling strategies that has the following three dimensions: the main actor in the error-handling process (machine versus user), the purpose of the strategy (error prevention, discovery, or correction), and the use of different modalities of interaction. The requirements that different error-handling strategies have on different sets of interaction modalities are also discussed. The main aim of this work is to establish a classification that can serve as a tool for understanding how to develop more efficient and more robust multimodal human-machine interfaces.Keywords: recognition-based technology, multimodal interfaces, error-handling, taxonomy, interaction design, interaction robustness Recognition-based technologyMultimodal interaction refers to interaction with the virtual and physical environment through natural modes of communication such as speech, body gestures, handwriting, graphics, or gaze.Unlike keyboards and mice inputs, natural modes of communication usually are non-deterministic, and have to be "recognised" by a recognition system, before they can be passed on to an application.Recent developments in recognition-based technology (e.g. speech and gesture recognition) have opened a myriad of new possibilities for the design and implementation of multimodal applications.Handwriting recognisers, for example, are being used in personal digital assistants (e.g. Paragon's multilingual PenReader software for Pocket PC devices), and speech recognition has made its way into desktop machines (e.g. IBM's ViaVoice TM speech recognition engines). However, designing and

show abstract

“…Multimodal applications are now being built in different application domain including medecine [15], military [5] or telecommunication [12]. For example, lot of mobile phones offer two input modalities, a keyboard and a voice recognizer, to interact.…”

Section: Introductionmentioning

confidence: 99%

Test of the ICARE Platform Fusion Mechanism

Dupuy-Chessa¹,

Bousquet²,

Bouchet³

et al. 2006

Interactive Systems. Design, Specification, and Verification

View full text Add to dashboard Cite

Abstract. Multimodal interactive systems offer a flexibility of interaction that increases their complexity. ICARE is a component-based approach to specify and develop multimodal interfaces using a fusion mechanism in a modality independent way. ICARE being reused to produce several multimodal applications, we want to ensure the correctness of its fusion mechanism. So we validate it using a test architecture based on Java technologies. This paper presents our validation approach, its results, its advantges and its limits.

show abstract

Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

Cited by 275 publications

References 72 publications

The Interactivity of Ict Tools

The Interactivity of Ict Tools

Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

Test of the ICARE Platform Fusion Mechanism

Contact Info

Product

Resources

About