Interpretation of multimodal designation with imprecise gesture

Abstract

We are interested in multimodal systems that use the following modes and modalities: speech (and natural language) as input as well as output, gesture as input and visual as output using screen displays. The user exchanges with the system by gesture and/or oral statements in natural language. This exchange, encoded in the different modalities, carries the goal of the user and also the designation of objects (referents) needed to achieve this goal. The system must identify in a precise and non-ambiguous way the objects designated by the user. In this paper, our main concern is the multimodal designations, with possibly imprecise gesture, of objects in the visual context. In order to identify such a designation, we propose a solution which uses probabilities, knowledge about manipulated objects, and perceptive aspects (degree of salience) associated with these objects.

A model for multimodal representation and processing for reference resolution

“…This internal representation associates a vector to every object which contains: the display name on the screen (it can be different to the database name), color, form, size, and salience. These characteristics are used when designating an object by speech and/or by gesture (more details in [2,4]). The salience of a graphical object consists of its visual and contextual weight.…”

“…The salience of a graphical object consists of its visual and contextual weight. To determine these weights, we use a salience distribution algorithm to objects in the common visual context [2,4] which distinguishes two moments of salience uses: saliences initialization at the beginning of each dialog and saliences modification during interaction. An object with its vector is an element of P .…”

“…• We use also the algorithm proposed in [4] to resolve designation by oral jointly to gesture (example 1). This algorithm uses elements from three of languages of the model: L, T , and P .…”

“…ρT −G (((x1, y1), (x2, y2), ...)) = object(riverN ame, blue, ...) ρT −G is resolved using the algorithm concerned by designation with gesture. It takes into account P , the object types founded by ρL−G(stream), and designation probabilities (more details in [4]). …”