Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Jiang, Huaizu; Sun, Deqing; Jampani, Varan; Yang, Ming–Hsuan; Learned-Miller, Erik; Kautz, Jan

doi:10.1109/cvpr.2018.00938

Cited by 721 publications

(800 citation statements)

References 38 publications

Supporting

Mentioning

761

Contrasting

Order By: Relevance

“…Direct estimation of the high-resolution flows, F l and F r , is difficult as the high-resolution target frame, I t , is not available. Previous methods compute bidirectional flows between neighboring frames and then apply a linear equation to estimate the flow to an intermediate frame assuming linear motion [4], [5]. Instead, we propose a novel approach to compute these flows by utilizing the information in both the main and auxiliary frames in two stages.…”

Section: Alignmentmentioning

confidence: 99%

“…Note that, recent state-of-the-art video frame interpolation methods have either used visibility maps [4] or Fig. 8.…”

Section: Appearance Estimationmentioning

confidence: 99%

“…To address this problem, existing techniques use a standard high-resolution video at low frame rate as input and increase the frame rate by interpolating the frames in between [1], [2], [3], [4], [5], [6]. Although these methods produce high-quality results for simple cases, their ability to handle challenging cases with complex motion is limited.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Paliwal

Kalantari

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Slow motion videos are becoming increasingly popular, but capturing high-resolution videos at extremely high frame rates requires professional high-speed cameras. To mitigate this problem, current techniques increase the frame rate of standard videos through frame interpolation by assuming linear motion between the existing frames. While this assumption holds true for simple cases with small motion, in challenging cases the motion is usually complex and this assumption is no longer valid. Therefore, they typically produce results with unnatural motion in these challenging cases. In this paper, we address this problem using two video streams as the input; an auxiliary video with high frame rate and low spatial resolution, providing temporal information, in addition to the standard main video with low frame rate and high spatial resolution. We propose a two-stage deep learning system consisting of alignment and appearance estimation that reconstructs high resolution slow motion video from the hybrid video input. For alignment, we propose to use a set of pre-trained and trainable convolutional neural networks (CNNs) to compute the flows between the missing frame and the two existing frames of the main video by utilizing the content of the auxiliary video frames. We then warp the existing frames using the flows to produce a set of aligned frames. For appearance estimation, we propose to combine the aligned and auxiliary frames using a context and occlusion aware CNN. We train our model on a set of synthetically generated hybrid videos and show high-quality results on a wide range of test scenes. We further demonstrate the practicality of our approach by showing the performance of our system on two real dual camera setups with small baseline.

show abstract

Section: Alignmentmentioning

confidence: 99%

“…Note that, recent state-of-the-art video frame interpolation methods have either used visibility maps [4] or Fig. 8.…”

Section: Appearance Estimationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Paliwal

Kalantari

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Only encoding frames with a 2D CNN is insufficient to maintain the temporal consistency of a video. Conventionally, people have used optical flow from input source videos as guidance to enhance the temporal consistency of target videos in various video processing tasks such as denoising [20], superresolution [17], frame interpolation [21], and style transfer [10]. Our work incorporates the temporal consistency constraint of inpainted area by jointly generating images and flows with a new loss function.…”

Section: Related Workmentioning

confidence: 99%

An Internal Learning Approach to Video Inpainting

Zhang

Mai

Jin

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We propose a novel video inpainting algorithm that simultaneously hallucinates missing appearance and motion (optical flow) information, building upon the recent 'Deep Image Prior' (DIP) that exploits convolutional network architectures to enforce plausible texture in static images. In extending DIP to video we make two important contributions. First, we show that coherent video inpainting is possible without a priori training. We take a generative approach to inpainting based on internal (within-video) learning without reliance upon an external corpus of visual data to train a one-size-fits-all model for the large space of general videos. Second, we show that such a framework can jointly generate both appearance and flow, whilst exploiting these complementary modalities to ensure mutual consistency. We show that leveraging appearance statistics specific to each video achieves visually plausible results whilst handling the challenging problem of long-term consistency.

show abstract

“…Interpolation techniques have been widely used in lots of computer vision and robotics tasks, which can be classified into two categories, i.e., temporal interpolation [1], [8], [14] and spatial interpolation [10], [12], [26]. In video processing, video interpolation aims to temporally generate an intermediate frame using two consecutive frames.…”

Section: Introductionmentioning

confidence: 99%

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Liu

Liao

Lin

et al. 2020

Sensors

View full text Add to dashboard Cite

LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10Hz) and have been widely applied in the field of autonomous driving and UAV. However, the camera with a higher frequency (around 20Hz) has to be decreased so as to match with LiDAR in a multisensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of Li-DAR sensors. PLIN can generate temporally and spatially highquality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guided by consecutive sparse depth maps and motion relationship. We also propose a refined interpolation stage guided by the realistic scene. Using this coarse-to-fine cascade structure, our method can progressively perceive multi-modal information and generate accurate intermediate point clouds.To the best of our knowledge, this is the first deep framework for Pseudo-LiDAR point cloud interpolation, which shows appealing applications in navigation systems equipped with LiDAR and cameras. Experimental results demonstrate that PLIN achieves promising performance on the KITTI dataset, significantly outperforming the traditional interpolation method and the state-of-the-art video interpolation technique.

show abstract

Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Cited by 721 publications

References 38 publications

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

Deep Slow Motion Video Reconstruction With Hybrid Imaging System

An Internal Learning Approach to Video Inpainting

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Contact Info

Product

Resources

About