From hand-perspective visual information to grasp type probabilities

Han, Mo; Gunay, Sezen Yagmur; Yıldız, İlkay; Bonato, Paolo; Önal, Çağdaş D.; Padır, Taşkın; Schirner, Gunar; Erdoğmuş, Deni̇z

doi:10.1145/3316782.3316794

Cited by 9 publications

(8 citation statements)

References 28 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering the case of embedding the camera in the palm of a robotic hand (or prosthetic hand), the identified grasp types for the same object from different approaching orientations of hand may be different [30,31]. Therefore, in [30], instead of making absolute positive or negative predictions, Han et al utilized the priority of grasp type as the form of prediction result, and took the ranked lists of grasps as a new label form. Based on this, they constructed a probabilistic classifier using the Plackett-Luce model.…”

Section: Related Workmentioning

confidence: 99%

DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning

Zhang¹,

Huang²,

Jiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

The task of grasp pattern recognition aims to derive the applicable grasp types of an object according to the visual information. Current state-of-the-art methods ignore category information of objects which is crucial for grasp pattern recognition. This paper presents a novel dual-branch convolutional neural network (DcnnGrasp) to achieve joint learning of object category classification and grasp pattern recognition. DcnnGrasp takes object category classification as an auxiliary task to improve the effectiveness of grasp pattern recognition. Meanwhile, a new loss function called joint cross-entropy with an adaptive regularizer is derived through maximizing a posterior, which significantly improves the model performance. Besides, based on the new loss function, a training strategy is proposed to maximize the collaborative learning of the two tasks. The experiment was performed on five household objects datasets including the RGB-D Object dataset, Hit-GPRec dataset, Amsterdam library of object images (ALOI), Columbia University Image Library (COIL-100), and MeganePro dataset 1. The experimental results demonstrated that the proposed method can achieve competitive performance on grasp pattern recognition with several state-of-the-art methods. Specifically, our method even outperformed the second-best one by nearly 15% in terms of global accuracy for the case of testing a novel object on the RGB-D Object dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning

Zhang¹,

Huang²,

Jiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The data used in this work is based on [12]. The dataset consists of 4130 images, which were augmented from 413 hand-perspective images of 102 ordinary objects including office and daily supplies, utensils, and complex-shaped objects.…”

Section: Datasetmentioning

confidence: 99%

“…For instance, they need calibration pretty often; the unexpected electrode shifting could distort the EMG signals; muscle fatigue and/or limb disposition adversely affect the EMG patterns; and some amputees may lack critical muscles which EMG classification rely on. These insufficiencies have led researchers to use more sources of information to understand human intent [12]. With the rise of Convolutional Neural Networks [19,23,25,5,27,28,14], studies on using visual information as a source of information for the prosthetic hand have been conducted [7,4,9,8,26,12], which focus on classifying images into a grasp type.…”

Section: Introductionmentioning

confidence: 99%

“…These insufficiencies have led researchers to use more sources of information to understand human intent [12]. With the rise of Convolutional Neural Networks [19,23,25,5,27,28,14], studies on using visual information as a source of information for the prosthetic hand have been conducted [7,4,9,8,26,12], which focus on classifying images into a grasp type.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Creating a Deployable Grasp Type Probability Estimator for a Prosthetic Hand

Zandigohar,

Han,

Erdogmus

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

For lower arm amputees, prosthetic hands promise to restore most of physical interaction capabilities. This requires to accurately predict hand gestures capable of grabbing varying objects and execute them timely as intended by the user. Current approaches often rely on physiological signal inputs such as Electromyography (EMG) signal from residual limb muscles to infer the intended motion. However, limited signal quality, user diversity and high variability adversely affect the system robustness. Instead of solely relying on EMG signals, our work enables augmenting EMG intent inference with physical state probability through machine learning and computer vision method. To this end, we: (1) study state-of-the-art deep neural network architectures to select a performant source of knowledge transfer for the prosthetic hand, (2) use a dataset containing object images and probability distribution of grasp types as a new form of labeling where instead of using absolute values of zero and one as the conventional classification labels, our labels are a set of probabilities whose sum is 1. The proposed method generates probabilistic predictions which could be fused with EMG prediction of probabilities over grasps by using the visual information from the palm camera of a prosthetic hand. Our results demonstrate that InceptionV3 achieves highest accuracy with 0.95 angular similarity followed by 1.4 MobileNetV2 with 0.93 at ∼ 20% the amount of operations.

show abstract

“…An alternative use of eye-view and hand-view paired images is the use of the CNN to provide a probability distribution for all possible grasp types, based on the hand-view image [11]. Such a CNN could be further extended to process the videos during the approach to the object, in order to generate a sequence of grasp type probability distributions over time.…”

Section: The Usage Of the Visual Informationmentioning

confidence: 99%

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Han

Gunay

Schirner

et al. 2019

Intel Serv Robotics

Self Cite

View full text Add to dashboard Cite

Upper limb and hand functionality is critical to many activities of daily living, and the amputation of one can lead to significant functionality loss for individuals. From this perspective, advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user, but more importantly from the improved capability to infer human intent from multimodal sensor data to provide the robotic hand perception abilities regarding the operational context. Such multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors including electromyography and inertial measurement units. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control. In this paper, we present a dataset of this type that was gathered with the anticipation of cameras being built into prosthetic hands, and computer vision methods will need to assess this hand-view visual evidence in order to estimate human intent. Specifically, paired images from human eye-view and hand-view of various objects placed at different orientations have been captured at the initial state of grasping trials, followed by paired video, EMG and IMU from the arm of the human during a grasp, lift, put-down, and retract style trial structure. For each trial, based on eye-view images of the scene showing the hand and object on a table, multiple humans were asked to sort in decreasing order of preference, five grasp types appropriate for the object in its given configuration relative to the hand. The potential utility of paired eye-view and hand-view images was illustrated by training a convolutional neural network to process hand-view images in order to predict eye-view labels assigned by humans. KeywordsMultimodal dataset • Human grasp intent classification • Prosthetic hand • Eye and hand-view images • EMG • Convolutional neural network

show abstract

From hand-perspective visual information to grasp type probabilities

Cited by 9 publications

References 28 publications

DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning

DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning

Towards Creating a Deployable Grasp Type Probability Estimator for a Prosthetic Hand

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Contact Info

Product

Resources

About