Neural Video Compression Using GANs for Detail Synthesis and Propagation

Mentzer, Fabian; Agustsson, Eirikur; Ballé, Johannes; Minnen, David; Johnston, Nick; Toderici, George

doi:10.1007/978-3-031-19809-0_32

Cited by 28 publications

(33 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These latter metrics express distance in terms of the output of features from an Inception network. They have recently increased in importance for perceptual evaluation of picture quality as used [24]. In order to select the optimal value of Λ, the table reports performance for Copt under the effect of varying Λ on a log scale.…”

Section: Resultsmentioning

confidence: 99%

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Ramsook¹,

Kokaram²

2023

Preprint

View full text Add to dashboard Cite

Image based Deep Feature Quality Metrics (DFQMs) have been shown to better correlate with subjective perceptual scores over traditional metrics. The fundamental focus of these DFQMs is to exploit internal representations from a large scale classification network as the metric feature space. Previously, no attention has been given to the problem of identifying which layers are most perceptually relevant. In this paper we present a new method for selecting perceptually relevant layers from such a network, based on a neuroscience interpretation of layer behaviour. The selected layers are treated as a hyperparameter to the critic network in a W-GAN. The critic uses the output from these layers in the preliminary stages to extract perceptual information. A video enhancement network is trained adversarially with this critic. Our results show that the introduction of these selected features into the critic yields up to 10% (FID) and 15% (KID) performance increase against other critic networks that do not exploit the idea of optimised feature selection.

show abstract

Section: Resultsmentioning

confidence: 99%

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Ramsook¹,

Kokaram²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Hence, recent deep-framework methods [? ], [6]- [8], [37]- [51] propose to construct end-to-end DLVC frameworks. Although DLVC methods can leverage the benefits of endto-end learning strategy, these methods still remove temporal redundancy by only referring to one or limited neighboring frames, which limits its performance improvement.…”

Section: Related Workmentioning

confidence: 99%

Scene Matters: Model-based Deep Video Compression

Tang¹,

Zhang²,

Gai³

et al. 2023

Preprint

View full text Add to dashboard Cite

Video compression has always been a popular research area, where many traditional and deep video compression methods have been proposed. These methods typically rely on signal prediction theory to enhance compression performance by designing high efficient intra and inter prediction strategies and compressing video frames one by one. In this paper, we propose a novel model-based video compression (MVC) framework that regards scenes as the fundamental units for video sequences. Our proposed MVC directly models the intensity variation of the entire video sequence in one scene, seeking non-redundant representations instead of reducing redundancy through spatiotemporal predictions. To achieve this, we employ implicit neural representation (INR) as our basic modeling architecture. To improve the efficiency of video modeling, we first propose contextrelated spatial positional embedding (CRSPE) and frequency domain supervision (FDS) in spatial context enhancement. For temporal correlation capturing, we design the scene flow constrain mechanism (SFCM) and temporal contrastive loss (TCL). Extensive experimental results demonstrate that our method achieves up to a 20% bitrate reduction compared to the latest video coding standard H.266 and is more efficient in decoding than existing video coding strategies.

show abstract

“…Besides, Lu et al further presented an online encoder updating scheme [40] from the perspective of content adaptation and error propagation. Afterwards, a series of end-to-end video compression algorithms [41], [42], [43], [44], [45], [46], [47], [48] were put forward to improve the coding performance. More specifically, Hu et al [41] proposed an end-to-end video compression framework by converting the input video to the latent code representation.…”

Section: Learning-based Video Codingmentioning

confidence: 99%

“…More specifically, Hu et al [41] proposed an end-to-end video compression framework by converting the input video to the latent code representation. The relevant techniques of recurrent learning [42] and adversarial learning [45], [47] were also introduced in the end-to-end compression framework. These learning-based compression algorithms could achieve promising coding efficiency in universal scenes, while there is still room for improvement considering the specific application scenarios such as talking face videos.…”

Section: Learning-based Video Codingmentioning

confidence: 99%

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Chen

Wang

et al. 2022

2022 Data Compression Conference (DCC)

View full text Add to dashboard Cite

In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression and headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation and head translation), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the stateof-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse. The project page can be found at https://github.com/Berlin0610/Interactive Face Video Coding.

show abstract

Neural Video Compression Using GANs for Detail Synthesis and Propagation

Cited by 28 publications

References 23 publications

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic

Scene Matters: Model-based Deep Video Compression

Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression

Contact Info

Product

Resources

About