Learning Image and Video Compression Through Spatial-Temporal Energy Compaction

Abstract

Compression has been an important research topic for many decades, to produce a significant impact on data transmission and storage. Recent advances have shown a great potential of learning image and video compression. Inspired from related works, in this paper, we present an image compression architecture using a convolutional autoencoder, and then generalize image compression to video compression, by adding an interpolation loop into both encoder and decoder sides. Our basic idea is to realize spatial-temporal energy compaction in learning image and video compression. Thereby, we propose to add a spatial energy compaction-based penalty into loss function, to achieve higher image compression performance. Furthermore, based on temporal energy distribution, we propose to select the number of frames in one interpolation loop, adapting to the motion characteristics of video contents. Experimental results demonstrate that our proposed image compression outperforms the latest image compression standard with MS-SSIM quality metric, and provides higher performance compared with state-of-the-art learning compression methods at high bit rates, which benefits from our spatial energy compaction approach. Meanwhile, our proposed video compression approach with temporal energy compaction can significantly outperform MPEG-4 and is competitive with commonly used H.264. Both our image and video compression can produce more visually pleasant results than traditional standards.

Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement

“…Most recently, several end-to-end deep video compression methods have been proposed [8,7,38,9,22,13]. Specifically, Wu et al [38] proposed predicting frames by interpolation from reference frames, and the image compression network of [31] is applied to compress the residual.…”

“…In 2019, Lu et al [22] proposed the Deep Video Compression (DVC) method, in which optical flow is used to estimate the temporal motion, and two auto-encoders are employed to compress the motion and residual, respectively. Meanwhile, in [9], spatial-temporal energy compaction is added into the loss function to improve the performance of video compression. Later, Habibian et al [13] proposed the rate-distortion auto-encoder, which uses an autoregressive prior for video entropy coding.…”

“…We compare HLVC with the latest learned video compression methods. Among them, Habibian et al [13] and Cheng et al [9] are optimized for MS-SSIM. DVC [22] and Wu et al [38] are optimized for PSNR.…”

“…Recent studies in learned image compression, e.g., [2,3], show the great potential of deep learning for improving the rate-distortion performance. It is therefore not surprising to see increasing interest in compressing video with Deep Neural Networks (DNNs) [8,38,9,22,13]. For example, Lu et al [22] proposed using optical flow for motion compensation and applying auto-encoders to compress the flow and residual.…”

Instance-Level Future Motion Estimation in a Single Image Based on Ordinal Regression and Semi-Supervised Domain Adaptation

“…Such proactive, predictive perceptual capabilities enable us to take desired actions and avoid dangerous situations. To make computing machines achieve a similar level to the human perception, motion understanding and representation have been studied in many computer vision tasks such as optical flow [1]- [3], object tracking [4]- [6], action recognition [7], future frame prediction [8], video interpolation [9], and video compression [10]. However, most conventional techniques depend on temporal information from multiple consecutive frames to estimate motions.…”