Insights from Generative Modeling for Neural Video Compression

Abstract

While recent machine learning research has revealed connections between deep generative models such as VAEs and rate-distortion losses used in learned compression, most of this work has focused on images. In a similar spirit, we view recently proposed neural video coding algorithms through the lens of deep autoregressive and latent variable modeling. We present recent neural video codecs as instances of a generalized stochastic temporal autoregressive transform, and propose new avenues for further improvements inspired by normalizing flows and structured priors. We propose several architectures that yield state-of-the-art video compression performance on full-resolution video and discuss their tradeoffs and ablations. In particular, we propose (i) improved temporal autoregressive transforms, (ii) improved entropy models with structured and temporal dependencies, and (iii) variable bitrate versions of our algorithms. Since our improvements are compatible with a large class of existing models, we provide further evidence that the generative modeling viewpoint can advance the neural video coding field.

Diffusion Probabilistic Modeling for Video Generation

“…We conjecture that, since the predicted next frame may be already motioncompensated, the resulting residual is sparser and, hence, easier to capture by the diffusion model. Similar observations have been made in neural video compression [9,26].…”

“…Some of these models show impressive rate-distortion performance with hierarchical structures that separately encode the prediction and error residual. Although compression models have different goals from generative models, both benefit from predictive sequential priors [26]. Note, however, that these models are ill-suited for generation since compression models typically have a small spatio-temporal context and are constructed to preserve local information rather than to generalize [12].…”

“…The autoregressive transform in Equation ( 5) has also been adapted in a VAE model [27] as well as in neural video compression architectures [9][10][11]26]. These approaches separately compress latent variables that govern the next-frame prediction as well as frame residuals, therefore achieving state-of-the-art rate-distortion performance on high-resolution video content.…”

“…In this paper, we extend diffusion probabilistic models for stochastic video generation. Our ideas are inspired by the principles of predictive coding [24,25] and neural compression algorithms [26] and draw on the intuition that residual errors are easier to model than dense observations [27]. Our architecture relies on two prediction steps: first, we employ a deterministic convolutional RNN to deterministically predict the next frame conditioned on a sequence of frames.…”

Understanding Entropy Coding With Asymmetric Numeral Systems (ANS): a Statistician's Perspective

“…Digitization in business and science raises the demand for data storage, and the proliferation of remote working arrangements makes high-quality videoconferencing indispensable. Recently, novel compression methods that employ probabilistic machine-learning models have been shown to outperform more traditional compression methods for images and videos , Yang et al, 2020a, Agustsson et al, 2020, Yang et al, 2021. Machine learning provides new methods for the declarative task of expressing complex probabilistic models, which are essential for data compression (see Section 2.1).…”