Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions

Abstract

Video frame interpolation aims to synthesize one or multiple frames between two consecutive frames in a video. It has a wide range of applications including slow-motion video generation, frame-rate up-scaling and developing video codecs. Some older works tackled this problem by assuming per-pixel linear motion between video frames. However, objects often follow a non-linear motion pattern in the real domain and some recent methods attempt to model per-pixel motion by non-linear models (e.g., quadratic). A quadratic model can also be inaccurate, especially in the case of motion discontinuities over time (i.e. sudden jerks) and occlusions, where some of the flow information may be invalid or inaccurate.In our paper, we propose to approximate the per-pixel motion using a space-time convolution network that is able to adaptively select the motion model to be used. Specifically, we are able to softly switch between a linear and a quadratic model. Towards this end, we use an end-toend 3D CNN encoder-decoder architecture over bidirectional optical flows and occlusion maps to estimate the nonlinear motion model of each pixel. Further, a motion refinement module is employed to refine the non-linear motion and the interpolated frames are estimated by a simple warping of the neighboring frames with the estimated perpixel motion. Through a set of comprehensive experiments, we validate the effectiveness of our model and show that our method outperforms state-of-the-art algorithms on four datasets (Vimeo, DAVIS, HD and GoPro).

An intelligent optimization algorithm with a deep learning‐enabled block‐based motion estimation model

“…The naïve BM approach exploits all pixels of the candidate blocks that match (Sinusas & Duncan, 2022), but to best determine the motion between input image pairs, the computational complexity is high. To reduce this, alternative approaches utilize representative values, where the blocks are considered based on a sub-block mean, particular pattern, or obvious pixels (Dutta et al, 2022). Yet, as these techniques account for all the blocks of the input image pair from the intensity area, determining the effectual value that represents the block is complex.…”