Speech and Language Processing for Multimodal Human-Computer Interaction

Enabling effective design of multimodal interfaces for speech-to-speech translation system: An empirical study of longitudinal user behaviors over time and user strategies for coping with errors

“…For example, it was reported that a multimodal interface enhanced the overall system and user performance by judiciously adopting multiple modalities, as introduced in the studies (Oviatt et al, 2004;Deng et al, 2004). Thus, in addition to speech, we incorporated a GUI and a "push-to-talk" modality into our system design.…”

“…Notable user modeling work includes the design and evaluation of multimodal interfaces (Oviatt et al, 2004;Dybkjaer et al, 2004;Deng et al, 2004), analysis of user behaviors (Oviatt et al, 2004;Shin et al, 2002), probabilistic user models (Eckert et al, 1997;Zukerman and Albrech, 2001), utility-based models (Horvitz and Paek, 2001), knowledge-based models (Komatani et al, 2003), and user simulation (Levin et al, 2000;Eckert et al, 1997;Scheffler and Young, 2002). It should be noted that mediated interpersonal communication systems (e.g., S2S translation systems) have been used in a very limited way in this context.…”

“…Previous work in spoken dialog systems showed that cognitive load is reduced while using a multimodal interface in comparison with that of a speech-only interface (Oviatt, 2006;Oviatt et al, 2004). In addition, it was reported that multimodal interfaces significantly improved user experience (Deng et al, 2004). Furthermore speech-centric multimodal interfaces provide opportunities for enhanced usability and naturalness and are an increasingly important research direction (Dybkjaer et al, 2004;Flanagan, 2004).…”

Interactive visualisation techniques for dynamic speech transcription, correction and training

“…the recogniser is provided with the speech input to be transcribed, 2. the entire input is transcribed (probably with errors) and presented to the user, 3. the user starts making manual correction from the beginning, producing transcripts that will be (substantially) correct, 4. corrections to words that are missing from the system's vocabulary are dynamically added to the vocabulary, 5. corrections of mistakes other than out-of-vocabulary words are used for training the speech recogniser's spea-ker model, 6. the system re-transcribes the speech related to the parts that are not yet corrected by the user, 7. steps 3-6 are repeated until no more corrections are needed.…”

“…Transcription systems are wide and varied in their uses, which include dictation [23] as well as audio [8,7], multimedia [6,25,18] and meeting indexing applications [24,22]. However, since even the most sophisticated automatic speech recognition systems are far from perfect, transcriptions produced by their application often contain errors.…”