HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Abstract

Upper limb and hand functionality is critical to many activities of daily living, and the amputation of one can lead to significant functionality loss for individuals. From this perspective, advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user, but more importantly from the improved capability to infer human intent from multimodal sensor data to provide the robotic hand perception abilities regarding the operational context. Such multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors including electromyography and inertial measurement units. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control. In this paper, we present a dataset of this type that was gathered with the anticipation of cameras being built into prosthetic hands, and computer vision methods will need to assess this hand-view visual evidence in order to estimate human intent. Specifically, paired images from human eye-view and hand-view of various objects placed at different orientations have been captured at the initial state of grasping trials, followed by paired video, EMG and IMU from the arm of the human during a grasp, lift, put-down, and retract style trial structure. For each trial, based on eye-view images of the scene showing the hand and object on a table, multiple humans were asked to sort in decreasing order of preference, five grasp types appropriate for the object in its given configuration relative to the hand. The potential utility of paired eye-view and hand-view images was illustrated by training a convolutional neural network to process hand-view images in order to predict eye-view labels assigned by humans. KeywordsMultimodal dataset • Human grasp intent classification • Prosthetic hand • Eye and hand-view images • EMG • Convolutional neural network

Disentangled Adversarial Autoencoder for Subject-Invariant Physiological Feature Extraction

“…HCI enables users to communicate their physiological information with machines for help with manipulating external devices in a more reliable, robust and safe manner. Traditionally, the assessment of physiological activity (e.g., human stress level and mental status) was implemented by monitoring signals such as electroencephalography (EEG) [2] and electromyography (EMG) [3]. However, these measurements require either surface (non-invasive) or implanted (invasive) electrodes and frequent calibration, which increase system cost and decrease user comfort.…”

Hybrid FPGA–CPU-Based Architecture for Object Recognition in Visual Servoing of Arm Prosthesis

“…The module predicts the class of the frame. The frame-level score ( ŷ( f , c)) is calculated as shown in Equation (3).…”

“…To overcome the limitations of traditional control solely based on the electromyographic (EMG) activity of the remaining muscles, promising alternatives consider hybrid systems combining noninvasive motion capture and vision control [1,2]. They include camera vision modules that allow for recognition of the subject's intention to grasp an object and assist visual control of prosthetic arms for object reaching and grasping [3].…”

NetCut: Real-Time DNN Inference Using Layer Removal

“…2) Dataset: The HANDS dataset [19] is a collection of images from graspable objects used in daily life including office supplies, utensils and complex-shaped objects like toys, from the hand camera perspective and different orientations. The labels are probabilistic as opposed to the common onehot encoding because of the feasibility of grasping objects in multiple ways with different preferences.…”