Unsupervised Video Interpolation Using Cycle Consistency

Reda, Fitsum A.; Sun, Deqing; Dundar, Aysegul; Shoeybi, Mohammad; Liu, Guilin; Shih, Kevin J.; Tao, Andrew; Kautz, Jan; Catanzaro, Bryan

doi:10.1109/iccv.2019.00098

Cited by 87 publications

(31 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also include an application to the video prediction task. Several methods have been proposed for video prediction problem, which in high level can be categorized as flowbased methods where methods model motion transformation via flows [22], [29], [37], [39] and latent-representation based methods [11], [12], [19], [31], [42]. Flow-based methods find the pixel correspondences between frames to move them accordingly in future frames.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Dundar¹,

Shih²,

Garg³

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. Using a motion-based foreground assumption, this work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions in an unsupervised way, allowing the model to condition only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest when measured against ground-truth foreground masks. Furthermore, the rendered background quality is also improved as ill-suited landmarks are no longer forced to model this content. We demonstrate this improvement via improved image fidelity in a video-prediction task. Code is available at https://github.com/NVIDIA/UnsupervisedLandmarkLearning

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Dundar¹,

Shih²,

Garg³

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, dataset acquisition method based on cycleGAN was proposed to obtain more data in limited training datasets [19].…”

Section: Related Workmentioning

confidence: 99%

Enhanced Real-Time Intermediate Flow Estimation for Video Frame Interpolation

Kim

Choi

2021

JWE

View full text Add to dashboard Cite

Recently, the demand for high-quality video content has rapidly been increasing, led by the development of network technology and the growth in video streaming platforms. In particular, displays with a high refresh rate, such as 120 Hz, have become popular. However, the visual quality is only enhanced if the video stream is produced at the same high frame rate. For the high quality, conventional videos with a low frame rate should be converted into a high frame rate in real time. This paper introduces a bidirectional intermediate flow estimation method for real-time video frame interpolation. A bidirectional intermediate optical flow is directly estimated to predict an accurate intermediate frame. For real-time processing, multiple frames are interpolated with a single intermediate optical flow and parts of the network are implemented in 16-bit floating-point precision. Perceptual loss is also applied to improve the cognitive performance of the interpolated frames. The experimental results showed a high prediction accuracy of 35.54 dB on the Vimeo90K triplet benchmark dataset. The interpolation speed of 84 fps was achieved for 480p resolution.

show abstract

“…The networks considered in video frame generation include the convolutional neural networks (CNNs) [32,33], recurrent neural networks (RNNs), long short term memory networks (LSTM), and GANs. Niklaus et al [1] designed a fully-convolutional CNN and estimated pairs of 1D kernels for all pixels to produce the interpolated frame.…”

Section: Image Generation and Video Frame Generationmentioning

confidence: 99%

Efficient Video Frame Interpolation Using Generative Adversarial Networks

Tran

Yang

2020

Applied Sciences

View full text Add to dashboard Cite

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

show abstract

Unsupervised Video Interpolation Using Cycle Consistency

Cited by 87 publications

References 21 publications

Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Enhanced Real-Time Intermediate Flow Estimation for Video Frame Interpolation

Efficient Video Frame Interpolation Using Generative Adversarial Networks

Contact Info

Product

Resources

About