Wearable Finger Tracking and Cutaneous Haptic Interface with Soft Sensors for Multi-Fingered Virtual Manipulation

Abstract

Multi-Fingered haptics is imperative for truly immersive virtual reality experience, as many real-world tasks involve finger manipulation. One of the key lacking aspect for this is the absence of technologically and economically viable wearable haptic interfaces that can simultaneously track the finger/hand motions and display multi-degree-of-freedom (DOF) contact forces. In this paper, we propose a novel wearable cutaneous haptic interface (WCHI), which consists of 1) finger tracking modules (FTMs) to estimate complex multi-DOF finger and hand motion; and 2) cutaneous haptic modules (CHMs) to convey three-DOF contact force at the finger-tip. By opportunistically utilizing such different types of sensors as inertial measurement units, force sensitive resistor sensors, and soft sensors, the WCHI can track complex anatomically consistent multi-DOF finger motion while avoiding FTM-CHM electromagnetic interference possibly stemming from their collocation in the small form-factor interface; while also providing the direction and magnitude of three-DOF finger-tip contact force, the feedback of which can significantly enhance the precision of contact force generation against variability among users via their closed-loop control. Human subject study is performed with a virtual peg insertion task to show the importance of both the multi-DOF finger tracking and the three-DOF cutaneous haptic feedback for dexterous manipulation in virtual environment.

Wearable Haptic Device for Stiffness Rendering of Virtual Objects in Augmented Reality

“…(2) The two-point grasp task in our experiment is valid to measure the perception of stiffness quantitatively; however, it is a rather simple task among real finger-based interactions in VR/AR (e.g., peg-in-hole [20], throwing, or stacking of virtual objects [29]); thus, the efficacy of the stiffness using the proposed kinesthetic haptic feedback should be verified for many real-world applications in terms of task-performance or human perception. (3) The proposed haptic device can be integrated with existing wearable hand tracking systems (e.g., [20]), to implement the first wearable haptic interface applicable for general AR finger-based applications.…”

“…Note that the contact plate is placed on the intermediate phalanx of the finger, not on the distal phalanx. This is because we want to alleviate the issues of the tendon-driven system (e.g., hysteresis, slack) by shortening the length of the tendon, assuming that the distal joints of the fingers have musculoskeletal dependencies with the parental joints (i.e., motion synergy [20,21,33]).…”

“…We attach two MOCAP markers on the index/thumb finger-tips of the developed haptic glove and four MOCAP markers on the HMD. Note that, for real-world AR manipulation, the MOCAP system can be replaced with an affordable tracking algorithm (e.g., [20]), yet, the MOCAP system is employed here to minimize the effect of tracking error on the stiffness perception.…”

“…For instance, the bulky exoskeleton structure of kinesthetic devices [15,16] can severely occlude or distort the visual appearance of the real hand. Even for cutaneous haptic devices with smaller form-factor [17,20,21], the fingertips, which are the most crucial parts for object manipulation, are severely occluded. These kinesthetic or cutaneous devices can lead to severe degradation of user experience and task performance in AR; thus, research changing the wearing location of the cutaneous device from the fingertip to the proximal phalanx [7,22] or utilizing vibrotactile haptic devices [23] are performed for AR finger-based manipulation.…”

Design and experimental evaluation of a new modular underactuated multi-fingered robot hand

“…If the working mode changes, one has to replace the robot hand. The replacement process is very cumbersome, and the control system of the traditional robot hand is complicated because in order to determine the gripping posture many sensors need to acquire force, position, torque information and so forth [33][34][35][36][37][38]. As a result, applicable robot hands that can be adapted to variable configurations and working modes remain a challenge.…”