Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering

Abstract

In this paper, we present an approach to tactile pose estimation from the first touch for known objects. First, we create an object-agnostic map from real tactile observations to contact shapes. Next, for a new object with known geometry, we learn a tailored perception model completely in simulation. To do so, we simulate the contact shapes that a dense set of object poses would produce on the sensor. Then, given a new contact shape obtained from the sensor output, we match it against the pre-computed set using the object-specific embedding learned purely in simulation using contrastive learning. This results in a perception model that can localize objects from a single tactile observation. It also allows reasoning over pose distributions and including additional pose constraints coming from other perception systems or multiple contacts. We provide quantitative results for four objects. Our approach provides high accuracy pose estimations from distinctive tactile observations while regressing pose distributions to account for those contact shapes that could result from different object poses. We further extend and test our approach in multi-contact scenarios where several tactile sensors are simultaneously in contact with the object. Website: mcube.mit.edu/research/tactile_loc_first_touch.html

Using Collocated Vision and Tactile Sensors for Visual Servoing and Localization

“…This along with our choice of smaller and highly textured objects preempts easy comparison with recent learned localization approaches (e.g. [23], [24], [29] etc.). Instead, we present overall results of accuracy experiments below.…”

“…None of the above experiments were able to generate reliable correspondences between the object model and the tactile image which leads us to conclude that in the absence of good initial pose estimates from an external sensor or learned embeddings (e.g. [24]), the portion of the object observed by the GelSight is often not large enough for conventional feature matching algorithms to work.…”

“…The trained network was then used to generate height maps of the surface of an object and the height maps were registered to reconstruct the object surface. This work was extended by [24] where the authors used a renderer to generate and cache a large number of possible tactile signals of objects from a data set touching a tactile sensor (GelSlim in this case) in different orientations. An actual tactile signal corresponding to a particular object at a particular pose was then compared with the cache to retrieve candidate object and pose pairs and the network demonstrated in [23] was then used to generate a height map which was registered with the candidate object at the candidate pose to localize contact.…”

“…We observe that in the absence of good initial pose estimates from an external sensor or learned embeddings (e.g. [24]), the portion of the object observed by the GelSight is often not large enough for conventional feature matching algorithms to work. Additionally, the GelSight elastomer mechanically operates as a nonlinear low pass spatial filter and does not preserve the true shape of depth discontinuities.…”

Tactile Pose Estimation and Policy Learning for Unknown Object Manipulation

“…In [46], Sudho et al learns a tactile observation model that takes visual-based tactile images and uses a factor graph to estimate the relative pose of the sensor while the end-effector is performing planar pushing task. Bauza et al [3] proposes an approach to localize objects using a learned observation model that predicts the local shape of the object from visuo-tactile readings. Later, they match this shape to simulated local shapes with known poses using FilterReg algorithm.…”