Opisthenar

Abstract

We introduce a vision-based technique to recognize static hand poses and dynamic finger tapping gestures. Our approach employs a camera on the wrist, with a view of the opisthenar (back of the hand) area. We envisage such cameras being included in a wrist-worn device such as a smartwatch, fitness tracker or wristband. Indeed, selected off-the-shelf smartwatches now incorporate a built-in camera on the side for photography purposes. However, in this configuration, the fingers are occluded from the view of the camera. The oblique angle and placement of the camera make typical vision-based techniques difficult to adopt. Our alternative approach observes small movements and changes in the shape, tendons, skin and bones on the opisthenar area. We train deep neural networks to recognize both hand poses and dynamic finger tapping gestures. While this is a challenging configuration for sensing, we tested the recognition with a real-time user test and achieved a high recognition rate of 89.4% (static poses) and 67.5% (dynamic gestures). Our results further demonstrate that our approach can generalize across sessions and to new users. Namely, users can remove and replace the wrist-worn device while new users can employ a previously trained system, to a certain degree. We conclude by demonstrating three applications and suggest future avenues of work based on sensing the back of the hand.

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

“…Dove et al [209] surveyed to understand how design innovation is practiced in the ML domain in terms of user experience. [63], [70], [165], [174], [211] [124], [125], [128], [175], [217] [69], [112], [113], [118], [130], [131], [133], [135], [137], [140], [144], [152], [153], [155] [49], [57], [67], [98], [110], [114], [180], [192], [193], [203], [214], [215] [48], [59], [62], [75], [79], [111], [120], [156], [159], [185], [195], [197], [207], [218] [65], [83],…”

“…Sterman et al [129] developed a tool to visualize and model writing styles using Deep Learning. Opisthenar [130] is a Deep Learning based tool to recognize head poses and finger tappings. Zhang et al [131] propose several techniques to correct errors more efficiently when typing on mobile phone keyboards.…”

“…Given that the majority in those sectors are not AI experts developing AI systems for them requires us to investigate the human aspect of such systems. Our analysis shows that application domains have specifically targeted gaming [63,70,165,174,211], interactive technologies [69,112,113,118,130,131,[133][134][135]137,140,144,152,153,155,212,213], medicine [49,57,67,110,114,180,192,193,203,214,215], psychiatry [98], music [65,83,85,102,153], sports [126], dating [60], video production [84], assistive technologies [2,80,86,88,89,…”

Hand with Sensing Sphere: Body-Centered Spatial Interactions with a Hand-Worn Spherical Camera

“…Based on the ability to capture information in a large area, it has become common to use vision sensors in the form of wearable devices to achieve hand-based interactions. Such sensors have been mainly attached to locations around the hand, such as fingers [8,40] and wrists [23,25,41,42,49,56,60], to obtain a desirable field of view for seeing the hand shape without interfering with hand movement. Nevertheless, these solutions are specifically designed for detecting the hand posture and cannot be applicable for detecting the hand location.…”

“…For detecting hand posture, vision-based wearable devices have become the focus of many works due to recent advancements in computer vision. For example, Opisthenar is a wrist-worn camera used to classify the hand's posture by seeing the back of the hand [60]. CyclopsRing is a ring-style wearable fisheye camera device that can distinguish the hand postures by analyzing the unique view caused by different postures [8].…”