Review of Bioinspired Vision-Tactile Fusion Perception (VTFP): From Humans to Humanoids

Abstract

Humanoid robots are designed and expected to resemble humans in structure and behavior, showing increasing application potentials in various fields. Like their biological counterparts, their environmental perception ability is fundamental. In particular, the visual and tactile perception are the two main sensory modes that humanoids use to understand and interact with the environment. Vision-Tactile Fusion Perception (VTFP) has shown multiple possibilities for better sensing understanding in challenging conditions, causing new research interests and questions. The overlap between visual and tactile perception in humanoids is continually growing. This work has reviewed the current state of the art of VTFP. It starts with the physiological basis of biological vision and tactile systems as well as the VTFP mechanisms as inspirations for humanoid perception. Then, the bioinspired visual-tactile fusion systems for humanoids are reviewed as the emphasis. After the survey on the vision and tactile sensors of robots, seven currently publicly available VTFP datasets are introduced. They are the data sources for several studies on neural network-inspired fusion algorithms. Furthermore, the applications of VTFP on humanoids are summarized. Finally, the challenges and future work are discussed. This review aims to provide several references for further exploitation of VTFP and its applications on humanoids.

RFCT: Multimodal Sensing Enhances Grasping State Detection for Weak-Stiffness Targets

“…Visual and tactile senses are the two main sensory modalities used by humanoid animals to understand and interact with their environment [32,33]. Our aim is for robots to integrate visual and tactile senses in state-detection tasks [34,35].…”

Vision-Based Tactile Sensors in Precision Agriculture: Deep Learning Approaches, Applications, and Limitations

“…Tactile sensing provides granular, localized data, complemented by the expansive overview offered by vision, encompassing attributes like an object's holistic shape and hue [3]. The merging of these senses, known as Vision-Tactile Fusion Perception (VTFP), has revealed numerous avenues for improved sensory comprehension [4,5]. Analogously, in robotic systems, while vision serves as a primary data source, tactile feedback is indispensable for discerning attributes like weight, firmness, slippage,and texture.…”

“…Much like when humans shop for avocados, tactile examination, or feeling its firmness, is essential for a more accurate assessment. For agricultural robotic manipulation, vision and touch stand out as the primary means to perceive and engage with their surroundings [5]. The merging of VTFP has revealed numerous avenues for improved sensory comprehension [4] and the intersection of visual and tactile sensing is consistently expanding in various domains [6].…”

NU-Biped-4.5: A Lightweight and Low-Prototyping-Cost Full-Size Bipedal Robot

“…While its origins can be traced back to the 1970s with the pioneering contributions of Professor Ichiro Kato and his colleagues at Waseda University in Japan, who developed various prototypes of humanoid robots over a span of 50 years (e.g., WABIAN-2 [1] (Figure 1A), HRP-4 [2] (Figure 1G), and HRP-5P [3]), the field of humanoid robotics has only recently gained increasing relevance. This growth is driven by advancements in actuation and sensory systems technology [4], artificial general intelligence (AGI) [5], and adaptive control systems [6]. As a result, this typology of robots has emerged as an ideal platform to embed AGI systems in the physical world.…”