Conditional Encoder-Based Adaptive Deep Image Compression with Classification-Driven Semantic Awareness

Abstract

This paper proposes a new algorithm for adaptive deep image compression (DIC) that can compress images for different purposes or contexts at different rates. The algorithm can compress images with semantic awareness, which means classification-related semantic features are better protected in lossy image compression. It builds on the existing conditional encoder-based DIC method and adds two features: a model-based rate-distortion-classification-perception (RDCP) framework to control the trade-off between rate and performance for different contexts, and a mechanism to generate coding conditions based on image complexity and semantic importance. The algorithm outperforms the QMAP2021 benchmark on the ImageNet dataset. Over the tested rate range, it improves the classification accuracy by 11% and the perceptual quality by 12.4%, 32%, and 1.3% on average for NIQE, LPIPS, and FSIM metrics, respectively.

Quantization-Based Adaptive Deep Image Compression Using Semantic Information

“…The classification accuracy and PSNR correspond to the classification and distortion objectives, respectively. SSIM, FSIM, and LPIPS measure the similarity of the reconstructed image to the original image on different feature domains, and NIQE is a measure of perceptual degree [18], [42]. For model performance evaluation, 500 test images are randomly selected from 100 categories of the ImageNet dataset, with 5 images per category.…”

“…However, during DIC model training, only the distortion objective was optimized, resulting in an essentially reconstruction-oriented ROI coding method with limited classification accuracy. To further improve the classification accuracy, the authors in [42] proposed a RDCP joint optimization framework to train the neural network for hybrid contexts. Averaged over the tested rate range, it outperforms [41] in classification accuracy, NIQE, LPIPS, and FSIM by 11%, 12.4%, 32%, and 1.3%, respectively.…”

“…The weights on loss terms have a global and implicit impact on the RDCP tradeoff. Previous works [18], [42] have investigated how the DCP metrics vary with changing λ 1 , λ SFM , and λ GAN . The distortion, classification, and perception objectives are mainly determined by the weights of λ 1 , λ SFM , and λ GAN .…”

“…Moreover, setting the values of λ SFM and λ GAN can balance the RDCP trade-off. We set λ SFM and λ GAN based on the experiment results in [18] and [42]. Controlling the reconstruction quality of ROI is another way to balance the classification and distortion objectives, which is affected by the λ ROI and λ CE .…”

“…In this section, we compare the performance of the proposed DIC algorithms with state-of-the-art adaptive DIC algorithms. The comparison algorithms include BM-CHENG, QMAP2021 [41], HEVRC [42], and RMPGC [18]. BM-CHENG is an adaptive DIC for image reconstruction context, while the rest are adaptive DIC algorithms for hybrid contexts.…”