Study of Multimodal Interfaces and the Improvements on Teleoperation

Abstract

Research in multimodal interfaces aims to provide immersive solutions and to increase overall human performance. A promising direction is to combine auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparison exists to show how combining audiovisuohaptic interfaces would affect human perception and by extent reflected on task performance. Our paper explores this idea and presents a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluated how each combination affects the performance in a study (N = 25) consisting of manipulation tasks with various difficulties. The overall performance was assessed using both subjective measures, by assessing cognitive workload and system usability, and objective measurements, by incorporating time and spatial accuracy-based metrics. The results showed that regardless of task complexity, the combination of stereoscopic-vision with the virtual reality headset increased performance across all measurements by 40%, compared to monocular-vision from a generic display monitor. Besides, using haptic feedback improved outcomes by 10% and auditory feedback accounted for approximately 5% improvement.

Electrotactile Feedback in Virtual Reality For Precise and Accurate Contact Rendering

“…V. DISCUSSION Our findings agree with previous studies, where bimodal feedback increased the accuracy and precision in contact tasks [3]. Also, our results are in accordance with a meta-analysis of user interaction studies, where the bimodal (i.e., visuohaptic) feedback was found to improve performance [19].…”

“…Rennes, IRISA, Inria -Rennes 35042, France. name.surname@irisa.fr 3 M. Marchal is with the Univ. Rennes, INSA, IRISA, Inria, CNRS -35042 Rennes, France and Institut Universitaire de France.…”

The Challenges in Modeling Human Performance in 3D Space with Fitts’ Law

“…One study investigated 3D virtual hand pointing with the index finger in addition to incorporating vibration feedback using a 3D screen display [63]. While disregarding depth and spatial arrangements, contrary to [5,59], they found that vibration feedback provides a reasonable addition to visual feedback, which appears to be in line with existing literature on multi-modal interfaces [2,60,82]. This is furthermore confirmed by numerous neuroscience studies indeed confirming that the simultaneous presence of both visual and somatosensory sensory cues is beneficial, particularly due to both modalities overlapping in the same brain region [1,3,36,73].…”

“…However, during object interaction, be it pointing or manipulation, rotation is a fundamental part. When performing a task that requires some kind of spatial accuracy, we humans usually attempt to match the rotation of the object so it satisfies certain spatial criteria [82]. Stoelen and Akin [77], were the first to combine both translational and rotational movements in one experiment.…”

“…Fitts' law is the most widely used human performance model in HCI history [4,17,24,31,40,52,76]. With the growth of networking and mixed reality technologies, the necessity of assessing human performance with a higher dimensional model has increased significantly [17,82]. Fitts' proposed his original model, known as simply Fitts' model in 1954 [19].…”