Multimodal Sensing Interface for Haptic Interaction

Abstract

This paper investigates the integration of a multimodal sensing system for exploring limits of vibrato tactile haptic feedback when interacting with 3D representation of real objects. In this study, the spatial locations of the objects are mapped to the work volume of the user using a Kinect sensor. The position of the user’s hand is obtained using the marker-based visual processing. The depth information is used to build a vibrotactile map on a haptic glove enhanced with vibration motors. The users can perceive the location and dimension of remote objects by moving their hand inside a scanning region. A marker detection camera provides the location and orientation of the user’s hand (glove) to map the corresponding tactile message. A preliminary study was conducted to explore how different users can perceive such haptic experiences. Factors such as total number of objects detected, object separation resolution, and dimension-based and shape-based discrimination were evaluated. The preliminary results showed that the localization and counting of objects can be attained with a high degree of success. The users were able to classify groups of objects of different dimensions based on the perceived haptic feedback.

Multimodal Navigation Systems for Users with Visual Impairments—A Review and Analysis

“…The usage of multimodal interfaces in navigational systems allows users to interact with systems using several communication modes. Diaz and Payandeh [71] argue that multimodal interfaces enable powerful, flexible and feature-rich interactive experiences.…”

Hand Motion and Posture Recognition in a Network of Calibrated Cameras

“…For hand gesture recognition, since the hand is usually the closest moving object to the camera, a suitable threshold can be defined on gray-scale images in order to eliminate any background noises. IR camera combined with retroreflective markers is also used in order to determine the position and posture of the hand in view of a single camera [21]. In this paper, two calibrated RGB cameras are employed to record hand gestures.…”

Finger and Palm Dynamic Pressure Monitoring for Basketball Shooting

“…Furthermore, our verified approach measuring the dynamic pressure distribution of the fingers and palm can be applied in many biomedical and robotic applications for simulating hand manipulation. For example, our approach can be modified to design an anthropomorphic robot hand and measure the pressure distributions of the robot hand [23], to achieve haptic feedback (tactile and force) [24,25] in robotic surgery using the da Vinci robotic system [26], and to measure the hand-arm biodynamic response for real power tool handles [27]. Due to these, our approach not only can be used for current sport-science applications, but also can provide an important design reference for developing future hand-manipulation stimulating technologies in medical and robotic applications.…”