Object recognition combining vision and touch

Abstract

This paper explores ways of combining vision and touch for the purpose of object recognition. In particular, it focuses on scenarios when there are few tactile training samples (as these are usually costly to obtain) and when vision is artificially impaired. Whilst machine vision is a widely studied field, and machine touch has received some attention recently, the fusion of both modalities remains a relatively unexplored area. It has been suggested that, in the human brain, there exist shared multi-sensorial representations of objects. This provides robustness when one or more senses are absent or unreliable. Modern robotics systems can benefit from multi-sensorial input, in particular in contexts where one or more of the sensors perform poorly. In this paper, a recently proposed tactile recognition model was extended by integrating a simple vision system in three different ways: vector concatenation (vision feature vector and tactile feature vector), object label posterior averaging and object label posterior product. A comparison is drawn in terms of overall accuracy of recognition and in terms of how quickly (number of training samples) learning occurs. The conclusions reached are: (1) the most accurate system is “posterior product”, (2) multi-modal recognition has higher accuracy to either modality alone if all visual and tactile training data are pooled together, and (3) in the case of visual impairment, multi-modal recognition “learns faster”, i.e. requires fewer training samples to achieve the same accuracy as either other modality.

Vision-Based Tactile Sensor Mechanism for the Estimation of Contact Position and Force Distribution Using Deep Learning

“…Yang et al [ 29 ] focused on analyzing the texture of the material using the micro RGB camera in the context of tactile finger instrumentation. A few studies, such as Corradi et al [ 30 ] and Luo et al [ 31 ], used the vision-based tactile mechanism to recognize various objects. There were also a few remarkable studies by Piacenza et al [ 32 ] which accurately estimated the contact position with indentation depth prediction using visual-tactile sensors.…”

Multisensory inclusive design with sensory substitution

“…In recent years, the concepts of cross-modal cognition have spread into multiple disciplines, and examples of their applications can be found in neural networks, artificial intelligence and cognitive robotics (Corradi, Hall, & Iravani, 2017 ; Hawkins & Blakeslee, 2005 ; Li, Zhu, Tedrake, & Torralba, 2019 ; Di Nuovo & Cangelosi, 2015 ). Now there are new opportunities to make use of sensory substitution techniques in a similar way.…”

Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

“…Similar studies related to the visual-tactile force sensor have been performed (e.g. [14,15]). However, the advantage of our proposed tactile sensor compared with similar studies is to use the position information of the virtual point markers instead of that of measured point markers.…”