Cross-modal visuo-tactile object recognition using robotic active exploration

Abstract

In this work, we propose a framework to deal with cross-modal visuo-tactile object recognition. By crossmodal visuo-tactile object recognition, we mean that the object recognition algorithm is trained only with visual data and is able to recognize objects leveraging only tactile perception. The proposed cross-modal framework is constituted by three main elements. The first is a unified representation of visual and tactile data, which is suitable for cross-modal perception. The second is a set of features able to encode the chosen representation for classification applications. The third is a supervised learning algorithm, which takes advantage of the chosen descriptor. In order to show the results of our approach, we performed experiments with 15 objects common in domestic and industrial environments. Moreover, we compare the performance of the proposed framework with the performance of 10 humans in a simple cross-modal recognition task.

A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Exploration

“…The accuracy is defined as the number of correct classification operations over the total number of classification operations. We compare different state-of-the-art descriptors with the proposed CLUE, as shown in Table I and Table II case study, we adopt the Cross Modal Recognition process (CMR) proposed in [3]. In order to show how strongly the preprocessing step described in Sec.…”

“…This paper presents how we tackle these challenges by supporting our proposed choices with a large set of experiments. The preliminary version of the proposed approach is presented in [3] and extended by leveraging results from transfer learning [4] to further improve the performance. With respect to the work in [3], the novel additional research questions are • do transfer learning approaches help to improve the performance of cross-modal visuo-tactile problems?…”

“…The preliminary version of the proposed approach is presented in [3] and extended by leveraging results from transfer learning [4] to further improve the performance. With respect to the work in [3], the novel additional research questions are • do transfer learning approaches help to improve the performance of cross-modal visuo-tactile problems? • are transfer learning approaches alternative to the solution proposed in [3] or can they be effectively combined?…”

“…With respect to the work in [3], the novel additional research questions are • do transfer learning approaches help to improve the performance of cross-modal visuo-tactile problems? • are transfer learning approaches alternative to the solution proposed in [3] or can they be effectively combined? The rest of the paper is organized as follows.…”

A Brief Survey on the Role of Dimensionality Reduction in Manipulation Learning and Control

“…A promising example consists of combining visual and tactile data. Cross-modal perception, on the other hand, consists in acquiring knowledge with a sensing modality and in reusing such a knowledge with a different modality [94]. For example, the robot can acquire knowledge from visual data and exploit the acquired knowledge when using tactile perception (cross-modal visuo-tactile perception), i.e.…”