Sandwiched Image Compression: Wrapping Neural Networks Around A Standard Codec

Abstract

We sandwich a standard image codec between two neural networks: a preprocessor that outputs neural codes, and a postprocessor that reconstructs the image. The neural codes are compressed as ordinary images by the standard codec. Using differentiable proxies for both rate and distortion, we develop a rate-distortion optimization framework that trains the networks to generate neural codes that are efficiently compressible as images. This architecture not only improves ratedistortion performance for ordinary RGB images, but also enables efficient compression of alternative image types (such as normal maps of computer graphics) using standard image codecs. Results demonstrate the effectiveness and flexibility of neural processing in mapping a variety of input data modalities to the rigid structure of standard codecs. A surprising result is that the rate-distortion-optimized neural processing seamlessly learns to transport color images using a single-channel (grayscale) codec.

Differentiable bit-rate estimation for neural-based video codec enhancement

“…The performance of the differentiable estimator of eq. ( 1) [8], on the other hand, is significantly less accurate (note that about 8% of the ratios Bit-rate estimation is easier in high-rate settings, and this can be observed by measuring the standard deviation of ratios measured / actual bit-rates, for different QP values, as shown in Table I. We can observe that, as the average bit rate varies by about one order of magnitude, the general pattern is the same observed in Fig.…”

“…One simple way to estimate bit-rates is to sum percoefficient estimates. For example, the differentiable approximation to ρ-domain estimation used in [8] is…”

“…Differentiable approximations have been developed for solving the problem that quantization derivatives are zero nearly everywhere [7], [8], [22]. While they are quite useful for distortion estimates, it is much harder to employ them in complex entropy coding processes based on quantized values.…”

“…For distortion estimation, the approaches developed for training end-to-end neural codecs (EENCs) provide good * Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. differentiable approximations, and can be used together with methods to estimate subjective quality [8], [9], [12].…”

HoloKinect: Holographic 3D Video Conferencing

“…Additionally, since losses are inevitable in the depth and video encoding process, we may be able to lessen these losses by pre-distorting the inputs. One could envision a lightweight neural network implementation constructed to minimize end-to-end errors in the depth maps with prior knowledge of the encoding and compression methodology, similar to the work done in [ 36 ].…”