Geometry-based next frame prediction from monocular video

Abstract

Abstract-We consider the problem of next frame prediction from video input. A recurrent convolutional neural network is trained to predict depth from monocular video input, which, along with the current video image and the camera trajectory, can then be used to compute the next frame. Unlike prior nextframe prediction approaches, we take advantage of the scene geometry and use the predicted depth for generating the next frame prediction. Our approach can produce rich next frame predictions which include depth information attached to each pixel. Another novel aspect of our approach is that it predicts depth from a sequence of images (e.g. in a video), rather than from a single still image.We evaluate the proposed approach on the KITTI dataset, a standard dataset for benchmarking tasks relevant to autonomous driving. The proposed method produces results which are visually and numerically superior to existing methods that directly predict the next frame. We show that the accuracy of depth prediction improves as more prior frames are considered.

Enlargement of the Field of View Based on Image Region Prediction Using Thermal Videos

“…In some earlier studies [23][24][25], a next frame was predicted using the current frame and previous sequential frames. A dual-motion GAN model (ConvLSTMGAN) was proposed [23], and image prediction was performed using a visible light image.…”

“…In this method, the proposed network was trained in a hybrid way using real and synthetic videos. In [25], a method for generating the next frame using a visible light image and ConvLSTM was proposed. In this study, the depth image is predicted using a current image and camera The remainder of this study is organized as follows.…”

Human Pose Forecasting via Deep Markov Models

“…Structured random forests [28], feed-forward CNNs [29] and variational autoencoders [30] have all been used to predict dense pixel trajectories from single frames. Prediction of raw pixels-as opposed to pixel trajectories-has been attempted using ordinary feedforward networks, GANs [31], [32] and RNNs [33], [34].…”

Intelligent Calibration of Static FEA Computations Based on Terrestrial Laser Scanning Reference

“…LSTM is a widely applicable kind of RNN which contains feedback connections for both single data points and entire data sequences in deep learning [ 50 ]. The optimization task regarding accurate future image prediction has been a highlighted problem in artificial intelligence in recent several years [ 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 ]. Kalchbrenner et al have developed a video pixel network to predict the joint distribution of future image in pixel videos [ 60 ].…”