Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Abstract

We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

Multimodal interaction with virtual worlds XMMVR: eXtensible language for MultiModal interaction with virtual reality worlds

“…The multimodal interaction architecture [14] used for developing a system allowing to flip a dual monitor in VR (Flip the monitor) and to change the color of a chair in AR is an architecture for multimodal interaction integrating voice recognition, gesture interaction and eye tracking to interact with a 3D agent. 2.…”

Distributed pointing for multimodal collaboration over sketched diagrams

“…Naturally occurring pointing gestures made from someone sitting across a table towards a single individual small element on a board sketch, say a milestone represented as a diamond shaped figure a few centimeters wide, would require a long time to be made, if it were to be unambiguously precise. Human beings overcome this difficulty by moving closer to the their targets, or by adding complementary information [17] via speech, disambiguating quick gestures towards an approximate region of focus by naming or describing the specific objects within this region [10].…”

“…Following MAVEN [10], the region of focus is expressed in terms of the intersection of a cone originating at the tip of the pointing arm and the surface of the interactive board. The angle of the cone is set based on the empirical evaluation (see Section 5) to indicate the pointing precision from a sitting position at a meeting table located at a distance from the interactive board.…”

“…The system is organized around a multiagent architecture, and is a direct descendant of the QuickSet [4] multimodal system and MAVEN [10], from which gesture recognition and multimodal integration components were taken. The system is an augmentation of the Charter Suite [9], to which collaboration support functionality is added as described in the present paper.…”

“…These rules were derived from an evaluation of characteristic patterns we identified after analyzing sensor profiles of the movements underlying the various gestures. The detailed gesture recognition mechanism was developed in the context of the Maven system [10].…”