Speech timing prediction in multimodal human-computer interaction

Abstract

This paper proposes a quantitative model of natural modality integration for speech and pointinggestures. An experiment is described that study temporal synchronization between speech and pointing gestures during multimodal interaction. The end of a pointing gesture (MT) is shown to be synchronized with either the key word of an expression or the deictic marker of a deictic expression. It is also shown that MT tends to be slightly behind the beginning of its associated word; this tendency becoming more marked for lengthy expressions (more than 6 syllables). All correlation scores were found to be above 0.9.Automatic understanding of natural multimodal interaction may advance only through good models of spontaneous communication. Specifically, the model proposed here can be efficiently used in the speech recognition system or in the interface management system to predict speech timing according to gesture timing.

Synchronization of speech and hand gestures during multimodal human-computer interaction

“…But all the proposed models are essentially of a qualitative nature. Moreover, very few studies have been done in the Human-Computer Interaction (HCI) domain [1] [3].…”

Using critical path analysis to model multimodal human–computer interaction

“…This does not necessarily mean that there will be no overlap in performance, e.g. it is possible that someone can begin to move the cursor on a drawing package and then speak a command before the cursor has reached its end-point (Bourguet & Ando, 1997). However, it does mean that MHCI is not simply a matter of combining two or more &unimodal' unit-tasks.…”

Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

“…For example, timing information from 3D hand pointing gestures has been used to automatically detect recognition errors in speech [53] [54]. Experimental studies have shown that, during speech and gesture multimodal interaction, 3D hand pointing gestures tend to be synchronised with either the nominal or the deictic ("this", "that", "here", etc.)…”

“…In [54], it is shown that the use of a speech and hand gestures synchronisation model can result in the recovery of up to a third of speech recognition errors.…”