Vid2Param: Modeling of Dynamics Parameters From Video

Abstract

Videos provide a rich source of information, but it is generally hard to extract dynamical parameters of interest. Inferring those parameters from a video stream would be beneficial for physical reasoning. Robots performing tasks in dynamic environments would benefit greatly from understanding the underlying environment motion, in order to make future predictions and to synthesize effective control policies that use this inductive bias. Online physical reasoning is therefore a fundamental requirement for robust autonomous agents. When the dynamics involves multiple modes (due to contacts or interactions between objects) and sensing must proceed directly from a rich sensory stream such as video, then traditional methods for system identification may not be well suited. We propose an approach wherein fast parameter estimation can be achieved directly from video. We integrate a physically based dynamics model with a recurrent variational autoencoder, by introducing an additional loss to enforce desired constraints. The model, which we call Vid2Param, can be trained entirely in simulation, in an end-to-end manner with domain randomization, to perform online system identification, and make probabilistic forward predictions of parameters of interest. This enables the resulting model to encode parameters such as position, velocity, restitution, air drag and other physical properties of the system. We illustrate the utility of this in physical experiments wherein a PR2 robot with a velocity constrained arm must intercept an unknown bouncing ball with partly occluded vision, by estimating the physical parameters of this ball directly from the video trace after the ball is released.

Future-Frame Prediction for Fast-Moving Objects with Motion Blur

“…Some studies were conducted to predict the future movement of balls rolling in elliptical bowls [8,32], and others were conducted to predict future object states and future frames by inputting short video sequences [9,33] using the spatial transform network [34]. A study was also conducted to predict the future trajectories of a bouncing ball [5] by using a variational recurrent neural network [35].…”

“…Some of these models were re-trained using real data to apply to real images [10]. Some other models were trained with synthetic data and applied directly to real data by generating synthetic data according to domain randomization techniques [5,40].…”

“…All the existing methods of processing real data introduced so far were designed to process only images or videos of clearly captured objects. Some of them added weak motion blur to the data [5,10], but did not deal with the data where severe motion blur was captured. Our proposed method can recognize the state of objects in the synthetic and real video sequences where the motion blur is severely captured.…”

“…To meet this requirement, several studies have been conducted in which a machine analyzes the physical properties of an object in an image and a video sequence using deep-learning-based methods. For example, studies have been conducted to infer the stability of stacked blocks from images [1][2][3] or to infer physical properties, such as mass or density, of objects from video sequences, where collisions occur between the objects [4,5]. Studies have also been conducted to predict future movements of objects [6,7] or predict physical properties and their changes [5,8,9] in a short video sequence.…”

“…However, in most of these studies, only videos of clearly captured objects have been focused upon. Although some studies have been conducted using video data with a small motion blur added to assume realistic video data [5,10], experiments were not performed with video data where a strong motion blur was captured.…”