Learned Video Compression

Abstract

We present a new algorithm for video coding, learned end-to-end for the low-latency mode. In this setting, our approach outperforms all existing video codecs across nearly the entire bitrate range. To our knowledge, this is the first ML-based method to do so.We evaluate our approach on standard video compression test sets of varying resolutions, and benchmark against all mainstream commercial codecs, in the low-latency mode. On standard-definition videos, relative to our algorithm, HEVC/H.265, AVC/H.264 and VP9 typically produce codes up to 60% larger. On high-definition 1080p videos, H.265 and VP9 typically produce codes up to 20% larger, and H.264 up to 35% larger. Furthermore, our approach does not suffer from blocking artifacts and pixelation, and thus produces videos that are more visually pleasing.We propose two main contributions. The first is a novel architecture for video compression, which (1) generalizes motion estimation to perform any learned compensation beyond simple translations, (2) rather than strictly relying on previously transmitted reference frames, maintains a state of arbitrary information learned by the model, and (3) enables jointly compressing all transmitted signals (such as optical flow and residual).Secondly, we present a framework for ML-based spatial rate control -a mechanism for assigning variable bitrates across space for each frame. This is a critical component for video coding, which to our knowledge had not been developed within a machine learning setting.

Video Compression With Rate-Distortion Autoencoders

“…Learned Video Compression Video compression shares many similarities with image compression, but the large size of video data, and the very high degree of redundancy create new challenges [15,30,33,40]. One of the first deep learning-based approaches proposes to model video autoregressively with a RNN-conditioned PixelCNN [23].…”

“…Very recently the video compression problem was attacked by considering flow compression and residual compression [27,33]. The additional components for flow and residual modeling allow to improve distortion in general, however, for low bit rates the proposed method is still outperformed by HEVC/H.265 on benchmark datasets.…”

“…Although much of this work has been motivated by potential future applications such as model based reinforcement learning, data compression is a very natural application that has received comparatively little attention. Deep learning-based video compression in particular has only recently started to be explored [11,33,40]. This is remarkable because improved video compression would * Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Figure 1: Overview of the proposed compression inference pipeline.…”

Deep Video Precoding

“…Adaptive spatio-temporal decomposition prior to encoding, followed by CNN-based spatio-temporal upscaling after decoding has been proposed by Afonso et al and was validated with H.265/HEVC encoding [31]. Finally, Wave One recently proposed video encoding with deep neural networks [32] and demonstrated quality gains against a conventional video encoder without B frames, and focusing on very-high bitrate encoding (20mbps or higher for FHD). While these are important achievements, most of these proposals are still outperformed by post-2013 video encoders, like HEVC and VP9, when utilized with their most advanced video buffering verifier (VBV) encoding configurations and appropriate constant rate factor tuning [33].…”

DSIC: Deep Stereo Image Compression

“…While we get to utilize shared information between two images, we are not able to exploit the spatial-temporal redundancies within a tightly coupled image sequence. There has been an abundance of work on traditional multi-view and stereo compression [12,14] as well as deep-learning based image and video compression [5,33,36,49]. However, the space of deep multi-view compression is relatively unexplored.…”