Inferring Interaction Force from Visual Information without Using Physical Force Sensors

Abstract

In this paper, we present an interaction force estimation method that uses visual information rather than that of a force sensor. Specifically, we propose a novel deep learning-based method utilizing only sequential images for estimating the interaction force against a target object, where the shape of the object is changed by an external force. The force applied to the target can be estimated by means of the visual shape changes. However, the shape differences in the images are not very clear. To address this problem, we formulate a recurrent neural network-based deep model with fully-connected layers, which models complex temporal dynamics from the visual representations. Extensive evaluations show that the proposed learning models successfully estimate the interaction forces using only the corresponding sequential images, in particular in the case of three objects made of different materials, a sponge, a PET bottle, a human arm, and a tube. The forces predicted by the proposed method are very similar to those measured by force sensors.

Dynamic-Vision-Based Force Measurements Using Convolutional Recurrent Neural Networks

“…Even though the changes in intensity obtained by subtracting frames, conventional cameras still suffer from low sampling rate as well as low dynamic range which limit the sensor performance. Moreover, LSTM-based networks are developed in [19] to estimate the contact force from sequential images on the soft objects. Similarly, CNNLSTM and 3D CNN networks are implemented in [20] to predict physical interactive force between to objects from a video.…”

Knocking and Listening: Learning Mechanical Impulse Response for Understanding Surface Characteristics

“…Therefore, RNNs were difficult to train through backpropagation. In recent approaches, existing simple neuronal structures have been modified using memory cells and gate units to more efficiently learn dependencies over longer intervals [35][36][37][38]. In this study, we evaluate the performance of the proposed floor type classification using two such neural networks, namely long short-term memory (LSTM) and gated recurrent units (GRUs).…”

Future-Frame Prediction for Fast-Moving Objects with Motion Blur

“…Models for inferring physical variables, such as mass and density, after observing a video of moving objects have been proposed [23][24][25]. Methods have also been proposed to infer the force applied to objects or the objects' current speed from images [26,27]. Some studies were conducted to predict future object states based on the current object states [28], making it possible to recognize the current state of an object from visual input and predicting subsequent frames [29].…”