Multi-Modal Geometric Learning for Grasping and Manipulation

Abstract

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

Beyond Top-Grasps Through Scene Completion

“…To date, many grasping methods rely fully or in part on deep learning. Some methods only use deep learning to extract additional information about objects with e.g., shape completion [9], [10] or tactile information [11] and then use analytical methods to plan the actual grasp [12], while others employ data-driven grasp planning in an endto-end fashion to generate grasps directly from images [1]- [8]. We will review both shape completion and end-to-end data-driven grasp planning as both are vital parts of our grasping pipeline.…”

“…In the context of shape completion from incomplete pointclouds, most recent improvements come from the adoption of deep learning. For instance, different works have explored tailored network structures [9], [13], [14], semantic object classification to aid the reconstruction [15], the integration of other sensing modalities such as tactile information [11], or the exploitation of the network uncertainty [10].…”

“…In the context of robotics grasping, [9], [11] and [10] are the most interesting as they not only focus on shape reconstruction quality but also on grasping accuracy. In this work, we make use of our previous shape completion network [10] to complete objects but instead of planning grasps with analytical methods-which is computationally expensive-we turn to data-driven grasp planning.…”

Multi-Modal Transfer Learning for Grasping Transparent and Specular Objects

“…Approaches to grasp synthesis can be classified into analytic and empirical methods; see Bohg et al [15] for a survey. Analytic approaches use physics-based contact models to compute force closure on an object, using the shape and estimated pose of the target object [16], [17], [18], but work poorly in the real world due to noisy sensing, simplified assumptions of contact physics, and difficulty in placing contact points accurately.…”

Trilateral convolutional neural network for 3D shape reconstruction of objects from a single depth view

“…Despite the fact that they successfully added context information to the voxel representations, their approach still required further improvements since the shape details of small 3D objects were missed. Later, Varley et al [3] presented a CNN approach for 3D shape reconstruction as part of a robot grasp planning algorithm from a single depth view [4]. This approach combined 3D convolutional layers with various fully connected layers to infer the complete 3D shape.…”