Interactive hand pose estimation using a stretch-sensing soft glove

Abstract

Fig. 1. Our stretch-sensing soft glove captures hand poses in real time and with high accuracy. It functions in diverse and challenging settings, like heavily occluded environments or changing light conditions, and lends itself to various applications. All images shown here are frames from recorded live sessions. We propose a stretch-sensing soft glove to interactively capture hand poses with high accuracy and without requiring an external optical setup. We demonstrate how our device can be fabricated and calibrated at low cost, using simple tools available in most fabrication labs. To reconstruct the pose from the capacitive sensors embedded in the glove, we propose a deep network architecture that exploits the spatial layout of the sensor itself. The network is trained only once, using an inexpensive off-the-shelf hand pose reconstruction system to gather the training data. The per-user calibration is then performed on-the-fly using only the glove. The glove's capabilities are demonstrated in a series of ablative experiments, exploring different models and calibration methods. Comparing against commercial data gloves, we achieve a 35% improvement in reconstruction accuracy.

A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods

“…To solve this problem, Glauser et al [40] extended the capacitive strain sensor concept of Atalay et al [38] to achieve dense area-stretch sensor arrays. Later, they designed a stretchable glove based on stretch array sensors, combined with a learned prior, to capture dense surface deformations of full hands [41] (Figure 4e).…”

“…Data glove made with stretch (strain) sensors. (a) A kinesthetic glove composed of five piezoresistive fabric (KPF) goniometers[33]; (b) wearable soft artificial sensing skin made of a hyperelastic elastomer material[36]; (c) data glove made of soft Ecoflex material[37]; (d) wearable glove based on highly stretchable textile-silicone capacitive sensors[38]; (e) glove made of a full soft composite of a stretchable capacitive silicone sensor array[41]. TA: Thumb Abduction; MM: Middle Metacarpal; RP: Ring Proximal; LM: Little Metacarpal; LA: Little Abduction.…”

Hand with Sensing Sphere: Body-Centered Spatial Interactions with a Hand-Worn Spherical Camera

“…The longest running approach to capturing precise 3D information regarding hand shape is to make use of glove-like devices (often referred to as data gloves) [17,26,63]. Although the estimation accuracy of such data gloves is high enough to be used for developing a broad range of interactions, wearing data glove often restricts the user's degrees-of-freedom and dulls the tactile sensitivity of the fingers.…”

DeepFisheye

“…Various approaches have been proposed for finger tracking. One of the most popular approaches is to use glove type de vices [11,16,19,30,42]. However, this approach is not appropriate for mobile interactions, because wearing gloves can be burdensome, especially in mobile situations.…”