Multi-Fingered In-Hand Manipulation With Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors

Abstract

Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize it for achieving manipulation skills is still an open issue. This paper presents a control method based on a graph convolutional network (GCN) which extracts geodesical features from the tactile data with complicated sensor alignments. Moreover, object property labels are provided to the GCN to adjust in-hand manipulation motions. Distributed tri-axial tactile sensors are mounted on the fingertips, finger phalanges and palm of an Allegro hand, resulting in 1152 tactile measurements. Training data is collected with a data-glove to transfer human dexterous manipulation directly to the robot hand. The GCN achieved high success rates for in-hand manipulation. We also confirmed that fragile objects were deformed less when correct object labels were provided to the GCN. When visualizing the activation of the GCN with a PCA, we verified that the network acquired geodesical features. Our method achieved stable manipulation even when an experimenter pulled a grasped object and for untrained objects. A Project page including accompanying video and supplementary materials can be found at https://sites.google.com/site/bashifunabashi/multifinger-project/in-hand-manipulation

TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network

“…Recent research from molecular biology, to multi-robot path planning [20] have demonstrated promising performance of GNNs to capture the interaction between a node in the graph for node and edge classification and message passing. This property yields the rising research in capturing tactile sensor data using GNN [12], [21] for in-hand manipulation. The properties of GNNs in capturing inter-node reactions inspired researchers to use GNN as a control algorithm.…”

“…Garcia-Garcia et al [21] proposed a GCN-based approach to capture taxel reading from BioTac SP taxel to binary classify grasps as stable or slippery ones. Funabashi et al [12] proposed a GCN-based pipeline to extract geodesical features from the tactile data from distributed taxels. However, their work does not consider how the topology of the graph changes as the taxel reading changes.…”

“…However, CNN accepts only rectangular inputs of predefined sizes, necessitating the resizing of tactile inputs which discards the inherent spatial relationship between taxels. To overcome this limitation, Graph Convolutional Network (GCN) [10] has been introduced to encode and interpret tactile information [11], [12]. Although GCN could handle unstructured tactile signals, the adjacency matrix of the graph, such as the number of nodes and connections between nodes must be specified from the start and remain fixed thereafter.…”

Tactile Transfer Learning and Object Recognition With a Multifingered Hand Using Morphology Specific Convolutional Neural Networks

“…Not only tactile information but also thermal information following robot configuration could be another application for our proposed methods and used for thermal imaging tasks [46], [47], [48]. As the proposed networks achieved high accuracies in recognition tasks, real-time recognition during multifingered manipulation is the next step for dexterous manipulation [49] using the graph convolutional network [50] inspired by the results of the MS-CNNs with morphology-related convolution. Moreover, transfer learning can be applied to more different domains, such as in-hand manipulation and measuring grasping stability.…”

Visuo-tactile pose tracking method for in-hand robot manipulation tasks of quotidian objects

“…One approach is the use of tactile sensing data in addition to visual information. Funabashi et al 12 developed a method that exploited a graphical convolutional network (GCN) to obtain geodesic characteristics of the tactile data from sensors placed all over the hand. This method enables in-hand manipulation and stable seizure of objects.…”