VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution

Chen, Zeyuan; Chen, Yinbo; Li, Jingwen; Xu, Xingqian; Goel, Vidit; Wang, Zhangyang; Wang, Xiaolong

doi:10.1109/cvpr52688.2022.00209

Cited by 46 publications

(21 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some unsupervised VFI models have been proposed in recent years, for example, Reda et al [28] developed a unsupervised VFI model based on cycle consistency. A more recent optical flow-based CNN model, VideoINR [24], successfully utilizes the implicit neural representation for continuous VFI.…”

Section: Related Workmentioning

confidence: 99%

“…Most of the previous models can only predict video frames with a fixed frame rate, which is defined by the training dataset. The only exceptions are the aforementioned Vid-ODE [16] and VideoINR [24]. VideoINR can only perform VFI and it does not satisfy the properties of NPs.…”

Section: Related Workmentioning

confidence: 99%

“…Both FFN and SIREN are efficient, and some work [19,20] have proven that they are equivalent to each other. INRs have been adopted for many computer vision tasks, including neural radiance fields for novel view synthesizing [21], image generation [22], unconditional video generation [23] and video interpolation [24].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Continuous conditional video synthesis by neural processes

Yang¹,

Bilodeau²

2022

Preprint

View full text Add to dashboard Cite

We propose a unified model for multiple conditional video synthesis tasks, including video prediction and video frame interpolation. We show that conditional video synthesis can be formulated as a neural process, which maps input spatio-temporal coordinates to target pixel values given context spatio-temporal coordinates and pixels values. Specifically, we feed an implicit neural representations of coordinates into a Transformer-based non-autoregressive conditional video synthesis model. Our task-specific models outperform previous work for video interpolation on multiple datasets and reach a competitive performance with the state-of-the-art models for video prediction. Importantly, the model is able to interpolate or predict with an arbitrary high frame rate, i.e., continuous synthesis. Our source code is available at https://github.com/NPVS/NPVS.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Continuous conditional video synthesis by neural processes

Yang¹,

Bilodeau²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that the FVD scores do not degrade even at higher spatial resolutions. Additionally, we compare INR-V with existing SOTA superresolution techniques Chen et al (2022) in Table 7 on 2048 videos randomly sampled from the RainbowJelly dataset. As can be seen, INR-V performs comparably with methods on the task of superresolution.…”

Section: A5 Inferring At Multiple Resolutions and Multiple Lengthsmentioning

confidence: 99%

INR-V: A Continuous Representation Space for Video-based Generative Tasks

Sen¹,

Agarwal²,

Namboodiri³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generating videos is a complex task that is accomplished by generating a set of temporally coherent images frame-by-frame. This limits the expressivity of videos to only image-based operations on the individual video frames needing network designs to obtain temporally coherent trajectories in the underlying image space. We propose INR-V, a video representation network that learns a continuous space for video-based generative tasks. INR-V parameterizes videos using implicit neural representations (INRs), a multi-layered perceptron that predicts an RGB value for each input pixel location of the video. The INR is predicted using a meta-network which is a hypernetwork trained on neural representations of multiple video instances. Later, the meta-network can be sampled to generate diverse novel videos enabling many downstream video-based generative tasks. Interestingly, we find that conditional regularization and progressive weight initialization play a crucial role in obtaining INR-V. The representation space learned by INR-V is more expressive than an image space showcasing many interesting properties not possible with the existing works. For instance, INR-V can smoothly interpolate intermediate videos between known video instances (such as intermediate identities, expressions, and poses in face videos). It can also in-paint missing portions in videos to recover temporally coherent full videos. In this work, we evaluate the space learned by INR-V on diverse generative tasks such as video interpolation, novel video generation, video inversion, and video inpainting against the existing baselines. INR-V significantly outperforms the baselines on several of these demonstrated tasks, clearly showcasing the potential of the proposed representation space.

show abstract

“…Specifically, TMNet [4] implements a temporal modulation network to interpolate arbitrary intermediate frame(s), but this kernel-based motion estimation method often results in serious temporal inconsistencies when dealing with large motions. The work most relevant to our approach is USTVSR [5] and VideoINR [6]. Both these two methods can modulate the input video to arbitrary resolution and frame rate, but they only consider the LR input within two neighboring video frames, and information from a long distance is ignored, which severely limiting their performance.…”

Section: Introductionmentioning

confidence: 99%

Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution

Wang

Han

Chen

2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

In this paper, we consider the task of space-time video super-resolution (ST-VSR), namely, expanding a given source video to a higher frame rate and resolution simultaneously. However, most existing schemes either consider a fixed intermediate time and scale in the training stage or only accept a preset number of input frames (e.g., two adjacent frames) that fails to exploit long-range temporal information. To address these problems, we propose a continuous ST-VSR (C-STVSR) method that can convert the given video to any frame rate and spatial resolution. To achieve time-arbitrary interpolation, we propose a forward warping guided frame synthesis module and an optical-flow-guided context consistency loss to better approximate extreme motion and preserve similar structures among input and prediction frames. In addition, we design a memory-friendly cascading depth-to-space module to realize continuous spatial upsampling. Meanwhile, with the sophisticated reorganization of optical flow, the proposed method is memory friendly, making it possible to propagate information from longrange neighboring frames and achieve better reconstruction quality. Extensive experiments show that the proposed algorithm has good flexibility and achieves better performance on various datasets compared with the state-of-the-art methods in both objective evaluations and subjective visual effects.

show abstract

VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution

Cited by 46 publications

References 37 publications

Continuous conditional video synthesis by neural processes

Continuous conditional video synthesis by neural processes

INR-V: A Continuous Representation Space for Video-based Generative Tasks

Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution

Contact Info

Product

Resources

About