On the Design and Development of Vision-based Tactile Sensors

Shah, Umer Hameed; Muthusamy, Rajkumar; Gan, Dongming; Zweiri, Yahya; Seneviratne, Lakmal

doi:10.1007/s10846-021-01431-0

Cited by 28 publications

(14 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonetheless, designing tactile sensors faced complexity in system integration and data processing, since increasing the scale requires a great deal of embedded sensing elements. Recently, vision-based tactile (ViTac) sensors have emerged as an effective method for the implementation of tactile sensing with a simple design [6], [7], [8]. In detail, the deformation of soft artificial skins upon physical contact with an object is detected through the optical tracking of visual features, such as markers or reflective membranes, which is then translated into tactile information, including contact location, force, vibration, object texture, and so on.…”

mentioning

confidence: 99%

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Luu

Nguyen

2023

IEEE Trans. Robot.

View full text Add to dashboard Cite

Large-scale robotic skin with tactile sensing ability is emerging with the potential for use in close-contact human-robot systems. Although recent developments in vision-based tactile sensing and related learning methods are promising, they have been mostly designed for small-scale use, such as by fingers and hands, in manipulation tasks. Moreover, learning perception for such tactile devices demands a huge tactile dataset, which complicates the data collection process. To address this, this study introduces a multiphysics simulation pipeline, called SimTacLS, which considers not only the mechanical properties of external physical contact but also the realistic rendering of tactile images in a simulation environment. The system utilizes the obtained simulation dataset, including virtual images and skin deformation, to train a tactile deep neural network to extract high-level tactile information. Moreover, we adopt a generative network to minimize sim2real inaccuracy, preserving the simulation-based tactile sensing performance. Last but not least, we showcase this sim2real sensing method for our large-scale tactile sensor (TacLink) by demonstrating its use in two trial cases, namely, whole-arm nonprehensile manipulation and intuitive motion guidance, using a custom-built tactile robot arm integrated with TacLink. This article opens new possibilities in the learning of transferable tactile-driven robotics tasks from virtual worlds to actual scenarios without compromising accuracy.

show abstract

mentioning

confidence: 99%

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Luu

Nguyen

2023

IEEE Trans. Robot.

View full text Add to dashboard Cite

show abstract

“…Due to not needing a source of illumination, a VBTS system that utilizes an event-based camera will have a lower instrumentation cost and thus require less maintenance in the long run. While some VBTSs use a transparent tactile surface to overcome the need for a source of illumination [3], this will make training end-toend machine learning models difficult as the camera will capture extraneous information from the environment making it dependent on the object the sensor is contacting and the environment, thus limiting generalization. Event-based cameras allow us to overcome the instrumentation and maintenance costs of having a source of illumination while still maintaining the potential for training end-to-end models.…”

Section: Neuromorphic Vision-based Tactile Sensingmentioning

confidence: 99%

“…as well as those based on piezoelectric material [2]. However, these sensors have high instrumentation costs and are thus hard to maintain over long periods of time [3]. A particularly promising tactile sensing technology is vision-based tactile sensors (VBTS) [3].…”

Section: Introduction 1sense Of Touch and Vision-based Tactile Sensingmentioning

confidence: 99%

TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing

Sajwani¹,

Ayyad²,

Alkendi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Vision-based tactile sensors (VBTS) have become the de facto method of giving robots the ability to obtain tactile feedback from their environment. Unlike other solutions to tactile sensing, VBTS offers high spatial resolution feedback without compromising on instrumentation costs or incurring additional maintenance expenses. However, conventional cameras used in VBTS have a fixed update rate and output redundant data, leading to computational overhead downstream. In this work, we present a neuromorphic vision-based tactile sensor (N-VBTS) that employs observations from an event-based camera for contact angle prediction. Particularly, we design and develop a novel graph neural network, dubbed TactiGraph, that asynchronously operates on graphs constructed from raw N-VBTS streams exploiting their spatiotemporal correlations to perform predictions. Although conventional VBTS uses an internal illumination source, TactiGraph is reported to perform efficiently in both scenarios, with and without an internal illumination source. Rigorous experimental results revealed that TactiGraph achieved a mean absolute error of 0.62∘ in predicting the contact angle and was faster and more efficient than both conventional VBTS and other N-VBTS, with lower instrumentation costs. Specifically, N-VBTS requires only 5.5% of the compute-time needed by VBTS when both are tested on the same scenario.

show abstract

“…In recent years, vision-based tactile sensors have drawn massive attention due to their easy development and economic maintenance (Shah et al, 2021 ). Such sensors use cameras to recognize the textures, features, or markers on a sensing skin and then compare their difference in task space with (or without) an illumination system to reconstruct the skin deformation, so as to acquire contact force information on the surface.…”

Section: Structural Evolvementsmentioning

confidence: 99%

A Survey of Multifingered Robotic Manipulation: Biological Results, Structural Evolvements, and Learning Methods

Wang

et al. 2022

Front. Neurorobot.

View full text Add to dashboard Cite

Multifingered robotic hands (usually referred to as dexterous hands) are designed to achieve human-level or human-like manipulations for robots or as prostheses for the disabled. The research dates back 30 years ago, yet, there remain great challenges to effectively design and control them due to their high dimensionality of configuration, frequently switched interaction modes, and various task generalization requirements. This article aims to give a brief overview of multifingered robotic manipulation from three aspects: a) the biological results, b) the structural evolvements, and c) the learning methods, and discuss potential future directions. First, we investigate the structure and principle of hand-centered visual sensing, tactile sensing, and motor control and related behavioral results. Then, we review several typical multifingered dexterous hands from task scenarios, actuation mechanisms, and in-hand sensors points. Third, we report the recent progress of various learning-based multifingered manipulation methods, including but not limited to reinforcement learning, imitation learning, and other sub-class methods. The article concludes with open issues and our thoughts on future directions.

show abstract

On the Design and Development of Vision-based Tactile Sensors

Cited by 28 publications

References 64 publications

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale

TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing

A Survey of Multifingered Robotic Manipulation: Biological Results, Structural Evolvements, and Learning Methods

Contact Info

Product

Resources

About