Stable In-Grasp Manipulation with a Low-Cost Robot Hand by Using 3-Axis Tactile Sensors with a CNN

Funabashi, Satoshi; Isobe, Tomoki; Ogasa, Shun; Ogata, Tetsuya; Schmitz, Alexander; Tomo, Tito Pradhono; Sugano, Shigeki

doi:10.1109/iros45743.2020.9341362

Cited by 14 publications

(10 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, the z-axis corresponds to the tactile information in the direction perpendicular to the sensor surface, and the other axes correspond to the tactile information in the tangential direction. This method of processing was used in [24], [30]. One way to process the tactile sensor patches is shown in Fig.…”

Section: B How To Input Tactile Information?mentioning

confidence: 99%

See 1 more Smart Citation

Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks

Funabashi

Yan

Hongyi

et al. 2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Multifingered robot hands can be extremely effective in physically exploring and recognizing objects, especially if they are extensively covered with distributed tactile sensors. Convolutional neural networks (CNNs) have been proven successful in processing high dimensional data, such as camera images, and are, therefore, very well suited to analyze distributed tactile information as well. However, a major challenge is to organize tactile inputs coming from different locations on the hand in a coherent structure that could leverage the computational properties of the CNN. Therefore, we introduce a morphology-specific CNN (MS-CNN), in which hierarchical convolutional layers are formed following the physical configuration of the tactile sensors on the robot. We equipped a four-fingered Allegro robot hand with several uSkin tactile sensors; overall, the hand is covered with 240 sensitive elements, each one measuring three-axis contact force. The MS-CNN layers process the tactile data hierarchically: at the level of small local clusters first, then each finger, and then the entire hand. We show experimentally that, after training, the robot hand can successfully recognize objects by a single touch, with a recognition rate of over 95%. Interestingly, the learned MS-CNN representation transfers well to novel tasks: by adding a limited amount of data about new objects, the network can recognize nine types of physical properties.

show abstract

Section: B How To Input Tactile Information?mentioning

confidence: 99%

“…CNNs have also been applied to tactile sensors for multifingered hands [23]. In our previous works, CNNs were also applied for in-hand manipulation and object recognition of an Allegro hand with three-axis tactile sensors [24], [25], and the results were better than the results of other modeling and machine learning methods.…”

mentioning

confidence: 99%

Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks

Funabashi

Yan

Hongyi

et al. 2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Fingertips with a human-like shape are beneficial for in-hand manipulation [23]. To fulfil these requirements, we cover the fingertips, phalanges and palm of an Allegro hand with uSkin tactile sensors, in a similar configuration to our previous works [4] [6].…”

Section: B Object Property and Processing Tactile Informationmentioning

confidence: 99%

“…In our past work, first, two-fingered manipulation was achieved using convolutional neural networks (CNNs) [6].…”

Section: Introductionmentioning

confidence: 99%

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Funabashi

Isobe

Hongyi

et al. 2022

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize it for achieving manipulation skills is still an open issue. This paper presents a control method based on a graph convolutional network (GCN) which extracts geodesical features from the tactile data with complicated sensor alignments. Moreover, object property labels are provided to the GCN to adjust in-hand manipulation motions. Distributed tri-axial tactile sensors are mounted on the fingertips, finger phalanges and palm of an Allegro hand, resulting in 1152 tactile measurements. Training data is collected with a data-glove to transfer human dexterous manipulation directly to the robot hand. The GCN achieved high success rates for in-hand manipulation. We also confirmed that fragile objects were deformed less when correct object labels were provided to the GCN. When visualizing the activation of the GCN with a PCA, we verified that the network acquired geodesical features. Our method achieved stable manipulation even when an experimenter pulled a grasped object and for untrained objects. A Project page including accompanying video and supplementary materials can be found at https://sites.google.com/site/bashifunabashi/multifinger-project/in-hand-manipulation

show abstract

“…Other recent works which use learning to process high-dimensional tactile data include [17] (using high frequency tactile information to achieve stable in-grasp manipulation for low-cost multi-fingered hands), [80] (predicting contact events such as slipping from tactile information to improve robustness of closed-loop grasping) and [71] (proposing a novel curriculum of action motion magnitude which makes learning more tractable).…”

Section: Learning For Tactile Sensingmentioning

confidence: 99%

Machine Learning for Robotic Manipulation

Vuong

2021

Preprint

View full text Add to dashboard Cite

The past decade has witnessed the tremendous successes of machine learning techniques in the supervised learning paradigm, where there is a clear demarcation between training and testing. In the supervised learning paradigm, learning is inherently passive, seeking to distill human-provided supervision in large-scale datasets into high capacity models. Following these successes, machine learning researchers have looked beyond this paradigm and became interested in tasks that are more dynamic. To them, robotics serve as an excellent test-bed, for the challenges of robotics break many of the assumptions that made supervised learning successful. Out of the many different areas within robotics, robotic manipulation has become a favorite area for researchers to demonstrate new algorithms because of the vast numbers of possible applications and its highly dynamical and complex nature. This document surveys recent robotics conferences and identifies the major trends with which machine learning techniques have been applied to the challenges of robotic manipulation.

show abstract

Stable In-Grasp Manipulation with a Low-Cost Robot Hand by Using 3-Axis Tactile Sensors with a CNN

Cited by 14 publications

References 30 publications

Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks

Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks

Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Machine Learning for Robotic Manipulation

Contact Info

Product

Resources

About