Cross-Modality Semantic Integration With Hypothesis Rescoring for Robust Interpretation of Multimodal User Interactions

Abstract

Abstract-We develop a framework pertaining to automatic semantic interpretation of multimodal user interactions using speech and pen gestures. The two input modalities abstract the user's intended message differently into input events, e.g., key terms/phrases in speech or different types of gestures in the pen modality. The proposed framework begins by generating partial interpretations for each input event as a ranked list of hypothesized semantics. We devise a cross-modality semantic integration procedure to align the pair of hypothesis lists between every speech input event and every pen input event in a multimodal expression. This is achieved by the Viterbi alignment algorithm that enforces the temporal ordering of the input events as well as the semantic compatibility of aligned events. The alignment enables generation of a unimodal, verbalized paraphrase that is semantically equivalent to the original multimodal expression. Our experiments are based on a multimodal corpus in the domain of city navigation. Application of the cross-modality integration procedure to near-perfect (manual) transcripts of the speech and pen modalities show that correct unimodal paraphrases are generated for over 97% of the training and test sets. However, if we replace with automatic speech and pen recognition transcripts, the performance drops to 53.7% and 54.8% for the training and test sets, respectively. In order to address this issue, we devised the hypothesis rescoring procedure that evaluates all candidates of cross-modality integration derived from multiple recognition hypotheses from each modality. The rescoring function incorporates the integration score, -best purity of recognized spoken locative expressions, as well as distances between coordinates of recognized pen gestures and their interpreted icons on the map. Application of cross-modality hypothesis rescoring improved the performance to 67.5% and 69.9% for the training and test sets, respectively. Index Terms-Joint integration, human-computer interaction, hypothesis rescoring, multimodal input, pen gesture, perplexity, robust interpretation, spoken input.

Latent Semantic Analysis for Multimodal User Input With Speech and Gestures

“…Speech recognition performance evaluated based on the top-scoring recognition hypotheses gave overall character accuracy of around 78%. 3 We have also developed a pen gesture recognizer based on a simple algorithm that proceeds through a sequential procedure of recognizing a point, a circle and a stroke [37]. This simple pen gesture recognition algorithm can generate -best output hypotheses.…”

“…This is used to indicate that there are too few or too many pen gestures aligned with one SLR or vice versa. The detailed explanation of this process of Viterbi alignment is provided in [37]. An illustrative example is shown in Fig.…”

Usage patterns and latent semantic analyses for task goal inference of multimodal user interactions

“…An SLR can be a direct (full name, abbreviated name or a contextual phrase such as "my current location") or an indirect one [11]. It may also be a singular, aggregated, plural reference or unspecified on number:  A singular reference can be a direct reference with a full name or an abbreviated name.…”

“…Previously, we have applied Belief Networks [12,13] for task goal inference based on unimodal (speech-only) inputs. However, previous studies [11,19] that compare the spoken part of multimodal inputs with unimodal (speech-only) inputs shows that the former generally has simpler syntactic structures, more diverse vocabularies and different term ordering. Therefore, we explore the use of latent semantic modeling (LSM) for task goal inference, with the objective of uncovering the associations between (unimodal or multimodal) terms and task goals through a data-derived latent space.…”

“…We leveraged such experiences to devise a computationally efficient approach based on score-based, cross-modal integration that incorporates semantic and temporal information [11]. The approach does not present high demands for training data.…”