Extending Neural P-frame Codecs for B-frame Coding

Pourreza, Reza; Cohen, Taco

doi:10.48550/arxiv.2104.00531

Cited by 1 publication

(1 citation statement)

References 29 publications

(59 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Video frame interpolation (VFI) (also known as Video temporal super-resolution) is a significant video enhancement problem which aims to synthesize one or more visually coherent frames between two consecutive frames in a video, i.e., to up-scale the number of video frames. Such an up-scaling method finds its usage in numerous video-based applications such as slow-motion video generation (e.g., in sports and TV commercials), video compressiondecompression framework [43], generating short videos from GIF images [48], novel view synthesis [12] and medical imaging [27,55].…”

Section: Introductionmentioning

confidence: 99%

Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions

Dutta¹,

Subramaniam²,

Mittal³

2022

Preprint

View full text Add to dashboard Cite

Video frame interpolation aims to synthesize one or multiple frames between two consecutive frames in a video. It has a wide range of applications including slow-motion video generation, frame-rate up-scaling and developing video codecs. Some older works tackled this problem by assuming per-pixel linear motion between video frames. However, objects often follow a non-linear motion pattern in the real domain and some recent methods attempt to model per-pixel motion by non-linear models (e.g., quadratic). A quadratic model can also be inaccurate, especially in the case of motion discontinuities over time (i.e. sudden jerks) and occlusions, where some of the flow information may be invalid or inaccurate.In our paper, we propose to approximate the per-pixel motion using a space-time convolution network that is able to adaptively select the motion model to be used. Specifically, we are able to softly switch between a linear and a quadratic model. Towards this end, we use an end-toend 3D CNN encoder-decoder architecture over bidirectional optical flows and occlusion maps to estimate the nonlinear motion model of each pixel. Further, a motion refinement module is employed to refine the non-linear motion and the interpolated frames are estimated by a simple warping of the neighboring frames with the estimated perpixel motion. Through a set of comprehensive experiments, we validate the effectiveness of our model and show that our method outperforms state-of-the-art algorithms on four datasets (Vimeo, DAVIS, HD and GoPro).

show abstract