Bayesian and Neural Inference on LSTM-Based Object Recognition From Tactile and Kinesthetic Information

Abstract

Recent advances in the field of intelligent robotic manipulation pursue providing robotic hands with touch sensitivity. Haptic perception encompasses the sensing modalities encountered in the sense of touch (e.g., tactile and kinesthetic sensations). This letter focuses on multimodal object recognition and proposes analytical and data-driven methodologies to fuse tactile-and kinesthetic-based classification results. The procedure is as follows: a three-finger actuated gripper with an integrated high-resolution tactile sensor performs squeeze-andrelease Exploratory Procedures (EPs). The tactile images and kinesthetic information acquired using angular sensors on the finger joints constitute the time-series datasets of interest. Each temporal dataset is fed to a Long Short-term Memory (LSTM) Neural Network, which is trained to classify in-hand objects. The LSTMs provide an estimation of the posterior probability of each object given the corresponding measurements, which after fusion allows to estimate the object through Bayesian and Neural inference approaches. An experiment with 36-classes is carried out to evaluate and compare the performance of the fused, tactile, and kinesthetic perception systems. The results show that the Bayesian-based classifiers improves capabilities for object recognition and outperforms the Neural-based approach.

Planning Multimodal Exploratory Actions for Online Robot Attribute Learning

“…To address this problem, several lines of research have shown that incorporating a variety of sensory modalities is the key to further enhance the robotic capabilities in recognizing multisensory object properties (see [4] and [21] for a review). For example, visual and physical interaction data yields more accurate haptic classification for objects [11], and non-visual sensory modalities (e.g., audio, haptics) coupled with exploratory actions (e.g., touch or grasp) have been shown useful for recognizing objects and their properties [5,10,15,24,30], as well as grounding natural language descriptors that people use to refer to objects [3,39]. More recently, researchers have developed end-to-end systems to enable robots to learn to perceive the environment and perform actions at the same time [20,42].…”

Recognition of Tactile Attribute Strength and Category Using Convolutional Neural Network

“…Therefore, some research will each image represents a transient moment pressure readings matrix, and haptic image sequence can contain the physical properties of a object information changes over time [20]. For haptic image, machine learning methods, such as k-nearest neighbor [21], bayesian method [22] and the traditional method based on image [23,24] was used to identify features. But this does not mean that haptic data must use these corresponding methods.…”

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

“…Researchers have also proposed to connect vision and touch via cross-domain modeling [13], [48], [14], [15] or estimating 3D human poses from a tactile carpet by taking the outputs of computer vision models as supervisions [49]. Others have tried to combine tactile sensing with proprioception and kinesthetic information for object and shape recognition [50], [51]. In this paper, we also desire to establish a connection between touch and vision but focusing on the task of dynamics modeling using the newlydeveloped tactile glove [1].…”