Event-Driven Visual-Tactile Sensing and Learning for Robots

Taunyazov, Tasbolat; Sng, Weicong; See, Hian Hian; Lim, Brian; Kuan, Jethro; Ansari, Abdul Fatir; Tee, Benjamin C. K.; Soh, Harold

doi:10.48550/arxiv.2009.07083

“…For instance, the GRID [164] visual-audio lipreading dataset is recorded using two bio-inspired silicon multimodal sensors (i.e., DVS [158] and DAS [159]). A visual-tactile event-based dataset [163] is built for intelligent power-efficient robot systems using a neuromorphic fingertip tactile sensor and an event camera. Besides, some works [150], [152]- [157], [165] attempt to integrate neuromorphic sensors with other sensing modalities (e.g., LiDAR, RGB-D camera, infrared camera, IMU, and GPS) for intelligent robots in challenging scenarios.…”

Section: B Categorizationmentioning

confidence: 99%

“…On the benchmark dataset side, existing BIC datasets can be classified into two categories: the simulated datasets [112], [134]- [143] and the real-world datasets [113], [114], [144]- [157] from the dataset generation perspective, and can also be classified into the single-modality datasets [144], [158]- [162] and the multi-modality datasets [150], [152]- [157], [163]- [165] from the modality perspective. The simulated BIC datasets are usually generated based on event-based simulators [134]- [140] or event cameras to record images from popular frame-based datasets on an LCD monitor [112], [141]- [143].…”

Section: Introduction Motivation and Overviewmentioning

confidence: 99%

Brain Inspired Computing: A Systematic Survey and Future Trends

Li¹,

Deng²,

Tang³

et al. 2023

Preprint

1

0

View full text Add to dashboard Cite

<p>Brain Inspired Computing (BIC) is an emerging research field that aims to build fundamental theories, models, hardware architectures, and application systems toward more general Artificial Intelligence (AI) by learning from the information processing mechanisms or structures/functions of biological nervous systems. It is regarded as one of the most promising research directions for future intelligent computing in the post-Moore era. In the past few years, various new schemes in this field have sprung up to explore more general AI. These works are quite divergent in the aspects of modeling/algorithm, software tool, hardware platform, and benchmark data, since BIC is an interdisciplinary field that consists of many different domains, including computational neuroscience, artificial intelligence, computer science, statistical physics, material science, microelectronics and so forth. This situation greatly impedes researchers from obtaining a clear picture and getting started in the right way. Hence, there is an urgent requirement to do a comprehensive survey in this field to help correctly recognize and analyze such bewildering methodologies. What are the key issues to enhance the development of BIC? What roles do the current mainstream technologies play in the general framework of BIC? Which techniques are truly useful in real-world applications? These questions largely remain open. To address the above issues, in this survey we first clarify the biggest challenge of BIC: how can AI models benefit from the recent advancements in computational neuroscience? With this challenge in mind, we will focus on discussing the concept of BIC and summarize four components of BIC infrastructure development: 1) modeling/algorithm; 2) hardware platform; 3) software tool; and 4) benchmark data. For each component, we will summarize its recent progress, main challenges to resolve, and future trends. On the basis of these studies, we present a general framework for the real-world applications of BIC systems, which is promising to benefit both AI and brain science. Finally, we claim that it is extremely important to build a research ecology to promote prosperity continuously in this field.</p>

show abstract

“…On the benchmark dataset side, existing BIC datasets can be classified into two categories: the simulated datasets [112], [134]- [143] and the real-world datasets [113], [114], [144]- [157] from the dataset generation perspective, and can also be classified into the single-modality datasets [144], [158]- [162] and the multi-modality datasets [150], [152]- [157], [163]- [165] from the modality perspective. The simulated BIC datasets are usually generated based on event-based simulators [134]- [140] or event cameras to record images from popular frame-based datasets on an LCD monitor [112], [141]- [143].…”

Section: Introduction Motivation and Overviewmentioning

confidence: 99%

Brain Inspired Computing: A Systematic Survey and Future Trends

Li¹,

Deng²,

Tang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Brain Inspired Computing (BIC) is an emerging research field that aims to build fundamental theories, models, hardware architectures, and application systems toward more general Artificial Intelligence (AI) by learning from the information processing mechanisms or structures/functions of biological nervous systems. It is regarded as one of the most promising research directions for future intelligent computing in the post-Moore era. In the past few years, various new schemes in this field have sprung up to explore more general AI. These works are quite divergent in the aspects of modeling/algorithm, software tool, hardware platform, and benchmark data, since BIC is an interdisciplinary field that consists of many different domains, including computational neuroscience, artificial intelligence, computer science, statistical physics, material science, microelectronics and so forth. This situation greatly impedes researchers from obtaining a clear picture and getting started in the right way. Hence, there is an urgent requirement to do a comprehensive survey in this field to help correctly recognize and analyze such bewildering methodologies. What are the key issues to enhance the development of BIC? What roles do the current mainstream technologies play in the general framework of BIC? Which techniques are truly useful in real-world applications? These questions largely remain open. To address the above issues, in this survey we first clarify the biggest challenge of BIC: how can AI models benefit from the recent advancements in computational neuroscience? With this challenge in mind, we will focus on discussing the concept of BIC and summarize four components of BIC infrastructure development: 1) modeling/algorithm; 2) hardware platform; 3) software tool; and 4) benchmark data. For each component, we will summarize its recent progress, main challenges to resolve, and future trends. On the basis of these studies, we present a general framework for the real-world applications of BIC systems, which is promising to benefit both AI and brain science. Finally, we claim that it is extremely important to build a research ecology to promote prosperity continuously in this field.</p>

show abstract

“…On the other hand, there has been a surge of interest and enthusiasm for learning-based systems that use deep learning to estimate tactile information [30]. Traditional image processing/computer vision-based and learning-based methods are both used for processing vision-based tactile sensors.…”

Section: Introductionmentioning

confidence: 99%

Design and development of a robust vision-based tactile sensor

Rayamane

¹

,

Ji

²

,

Packianather

³

2022

2022 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)

View full text Add to dashboard Cite

For robots to perform advanced manipulation of objects, touch is a critical source of information, and a highquality tactile sensor is essential. Image-based optical tactile sensors, and its inheritances, which have soft touch interfaces, can provide high-resolution tactile images of the contact geometry, contact pressure, and slip conditions. However, due to the lack of robustness provided by the current tactile sensors, the ability to grasp hard or sharp objects is minimal. In this work, we propose an image-based optical tactile sensor and overcome the above limitation of poor robustness by introducing a latex layer on the touch interface. We use a combination of silicone elastomer covered with a latex material and an acrylic sheet to support the silicone elastomer. A camera placed at the bottom of the sensor housing captures the deformation of the elastomer surface illuminated by an inner light. To evaluate the performance, we carried out a series of experiments. First, we evaluated the mechanical characteristics of the silicone elastomer with three types of coating, namely latex membrane, metallic coating, and no coating. The proposed latex membrane clearly outperformed the other two in terms of robustness. Second, we carried out the force-displacement experiments quantitatively to further study the sensitivity and robustness. Last, we validated the sensor performance in terms of its spatial resolution by applying the VGG-19 neural network for classifying touch patterns captured by the sensor. Overall, the proposed sensor achieved the desired robustness, sensitivity, and spatial resolution performance.

show abstract

Event-Driven Visual-Tactile Sensing and Learning for Robots

Cited by 11 publications

References 0 publications

Brain Inspired Computing: A Systematic Survey and Future Trends

Brain Inspired Computing: A Systematic Survey and Future Trends

Brain Inspired Computing: A Systematic Survey and Future Trends

Design and development of a robust vision-based tactile sensor

Contact Info

Product

Resources

About