Vision-Based Assistance for Myoelectric Hand Control

He, Yong; Kubozono, Ryusuke; Fukuda, Osamu; Yamaguchi, Nobuhiko; Okumura, Hiroshi

doi:10.1109/access.2020.3036115

Cited by 20 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, note that we hypothesize to have only one object in the scene. The extension to multiple objects would include a preliminary step of target object identification such as the ones proposed in [33], [20] which is out of the scope of this paper but can be considered as future work.…”

Section: Methodsmentioning

confidence: 99%

“…Conversely, the eye-in-hand configuration can be completely transparent to the user and allows to gather closer views of the objects to grasp. This makes it easier to identify the target with visual and motion cues [20]. For instance, in [21] the geometrical information (centroid and major axes) of the target object is inferred by using an RGB-D camera placed on the prosthesis and it is used to control the wrist orientation through visual servoing.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Federico¹,

Maiettini²,

Pasquale³

et al. 2022

Preprint

View full text Add to dashboard Cite

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-inhand systems automatically control the hand aperture and preshaping before the grasp, based on visual input coming from a camera on the wrist. In this work, we present an eye-inhand learning-based approach for hand pre-shape classification from RGB sequences. In order to reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories for the purpose. We tackle the peculiarity of the eye-in-hand setting by means of a model for the human arm trajectories, with domain randomization over relevant visual elements. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. Our code, real and synthetic datasets will be released upon acceptance.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Federico¹,

Maiettini²,

Pasquale³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The experiments showed good results in terms of object segmentation and classification accuracy. Similarly, in [24] an object detection network classifies different objects driving the selection of grasping patterns based on the object kind, without considering the actual grasping surface. The uncertainty introduced by partial occlusion and movements was explicitly modeled in [25].…”

Section: Related Workmentioning

confidence: 99%

Hardware-Aware Affordance Detection for Application in Portable Embedded Systems

Ragusa¹,

Gianoglio

Došen

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Affordance detection in computer vision allows segmenting an object into parts according to functions that those parts afford. Most solutions for affordance detection are developed in robotics using deep learning architectures that require substantial computing power. Therefore, these approaches are not convenient for application in embedded systems with limited resources. For instance, computer vision is used in smart prosthetic limbs, and in this context, affordance detection could be employed to determine the graspable segments of an object, which is a critical information for selecting a grasping strategy. This work proposes an affordance detection strategy based on hardware-aware deep learning solutions. Experimental results confirmed that the proposed solution achieves comparable accuracy with respect to the state-ofthe-art approaches. In addition, the model was implemented on real-time embedded devices obtaining a high FPS rate, with limited power consumption. Finally, the experimental assessment in realistic conditions demonstrated that the developed method is robust and reliable. As a major outcome, the paper proposes and characterizes the first complete embedded solution for affordance detection in embedded devices. Such a solution could be used to substantially improve computer vision based prosthesis control but it is also highly relevant for other applications (e.g., resource-constrained robotic systems).

show abstract

“…Our research group has also been developing a visionbased prosthetic hand based on deep learning technology [35] [36] [37] [38]. In [39], we designed a prosthetic hand control method that can determine the grasping target and motion according to the spatial and temporal relationship between the prosthetic hand and the objects, such as distance, position, and gazing time. The developed hand captures the environment with an onboard vision sensor and determines the object to be grasped; then, the motor is triggered by sEMG activation measured from the operator's skin surface.…”

Section: Related Workmentioning

confidence: 99%

Bimodal Control of a Vision-Based Myoelectric Hand

Fukuda

Sakaguchi

et al. 2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a novel control scheme for a vision-based prosthetic hand. To realize complex and flexible human-like hand movements, the proposed method fuses bimodal information. Combining information from surface EMG signals with object information from a vision sensor, the system can select an appropriate hand motion. The training/recognition using both sEMG signals and object images can be performed with a single deep neural network in an end-to-end manner. The bimodal sensor information enables the system to recognize the operator's intended motion with higher accuracy than that of the conventional method using only sEMG signals. In addition, the generalization ability of the network is improved, so motion recognition robustness is enhanced against abnormal data that include partly noisy or missing samples. To verify the validity of the proposed approach, we prepared a dataset that contains the sEMG signals and the object images for 10 types of grasping motions. Three kinds of experiments were conducted: comparison of the proposed method with the conventional method, examination of the recognition robustness against partly noisy or missing samples, and challenges to recognize hand motions based on raw sEMG signals. The results revealed that the proposed bimodal network achieved considerably high recognition performance.

show abstract

Vision-Based Assistance for Myoelectric Hand Control

Cited by 20 publications

References 27 publications

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Hardware-Aware Affordance Detection for Application in Portable Embedded Systems

Bimodal Control of a Vision-Based Myoelectric Hand

Contact Info

Product

Resources

About