Beyond Coding: Detection-driven Image Compression with Semantically Structured Bit-stream

He, Tianyu; Sun, Simeng; Guo, Zongyu; Chen, Zhibo

doi:10.1109/pcs48520.2019.8954525

Cited by 16 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The sensing matrix proposed in this paper is a simple process that retains certain set of pixels and forces other pixels zero. The image reconstruction from the reduced set of pixels is a CS problem [34][35][36] and the retained pixels can be considered as CS measurements. This process can be modeled as array multiplication of image and a binary mask.…”

Section: Proposed Sensing Matrix Learningmentioning

confidence: 99%

Compressive Domain Deep CNN for Image Classification and Performance Improvement Using Genetic Algorithm-Based Sensing Mask Learning

Ramachandran

2022

Applied Sciences

View full text Add to dashboard Cite

The majority of digital images are stored in compressed form. Generally, image classification using convolution neural network (CNN) is done in uncompressed form rather than compressed one. Training the CNN in the compressed domain eliminates the requirement for decompression process and results in improved efficiency, minimal storage, and lesser cost. Compressive sensing (CS) is one of the effective and efficient method for signal acquisition and recovery and CNN training on CS measurements makes the entire process compact. The most popular sensing phenomenon used in CS is based on image acquisition using single pixel camera (SPC) which has complex design implementation and usually a matrix simulation is used to represent the SPC process in numerical demonstration. The CS measurements using this phenomenon are visually different from the image and to add this in the training set of the compressed learning framework, there is a need for an inverse SPC process that is to be applied all through the training and testing dataset image samples. In this paper we proposed a simple sensing phenomenon which can be implemented using the image output of a standard digital camera by retaining few pixels and forcing the rest of the pixels to zero and this reduced set of pixels is assumed as CS measurements. This process is modeled by a binary mask application on the image and the resultant image still subjectively legible for human vision and can be used directly in the training dataset. This sensing mask has very few active pixels at arbitrary locations and there is a lot of scope to heuristically learn the sensing mask suitable for the dataset. Only very few attempts had been made to learn the sensing matrix and the sole effect of this learning process on the improvement of CNN model accuracy is not reported. We proposed to have an ablation approach to study how this sensing matrix learning improves the accuracy of the basic CNN architecture. We applied CS for two class image dataset by applying a Primitive Walsh Hadamard (PWH) binary mask function and performed the classification experiment using a basic CNN. By retaining arbitrary amount of pixel in the training and testing dataset we applied CNN on the compressed measurements to perform image classification and studied and reported the model performance in terms of training and validation accuracies by varying the amount of pixels retained. A novel Genetic Algorithm-based compressive learning (GACL) method is proposed to learn the PWH mask to optimize the model training accuracy by using two different crossover techniques. In the experiment conducted for the case of compression ratio (CR) 90% by retaining only 10% of the pixels in every images both in training and testing dataset that represent two classes, the training accuracy is improved from 67% to 85% by using diagonal crossover in offspring creation of GACL. The robustness of the method is examined by applying GACL for user defined multiclass dataset and achieved better CNN model accuracies. This work will bring out the strength of sensing matrix learning which can be integrated with advanced training models to minimize the amount of information that is to be sent to central servers and will be suitable for a typical IoT frame work.

show abstract

Section: Proposed Sensing Matrix Learningmentioning

confidence: 99%

Compressive Domain Deep CNN for Image Classification and Performance Improvement Using Genetic Algorithm-Based Sensing Mask Learning

Ramachandran

2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Moreover, different from conventional image coding, machine-orientated image compression only conveys necessary information for subsequent machine consumption, which is less expensive and of higher secure level. For example, a learning-based semantic structure coding (SSC) was proposed in [56], [57], where an object detection module in the feature domain is used to locate and align objects and then reorganize the features with such information for generate a semantically structured bitstream (SSB). As a result, it can directly obtain the information required by a variety of machine analysis tasks from the designated area in the SSB, instead of transmitting a complete bitstream.…”

Section: B Image Compression For Machinesmentioning

confidence: 99%

A New Image Codec Paradigm for Human and Machine Uses

Chen¹,

Jin²,

Meng³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the AI of Things (AIoT) development, a huge amount of visual data, e.g., images and videos, are produced in our daily work and life. These visual data are not only used for human viewing or understanding but also for machine analysis or decision-making, e.g., intelligent surveillance, automated vehicles, and many other smart city applications. To this end, a new image codec paradigm for both human and machine uses is proposed in this work. Firstly, the high-level instance segmentation map and the low-level signal features are extracted with neural networks. Then, the instance segmentation map is further represented as a profile with the proposed 16-bit gray-scale representation. After that, both 16-bit gray-scale profile and signal features are encoded with a lossless codec. Meanwhile, an image predictor is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features. Finally, the residual map between the original image and the predicted one is compressed with a lossy codec, used for high-quality image reconstruction. With such designs, on the one hand, we can achieve scalable image compression to meet the requirements of different human consumption; on the other hand, we can directly achieve several machine vision tasks at the decoder side with the decoded 16-bit gray-scale profile, e.g., object classification, detection, and segmentation. Experimental results show that the proposed codec achieves comparable results as most learningbased codecs and outperforms the traditional codecs (e.g., BPG and JPEG2000) in terms of PSNR and MS-SSIM for image reconstruction. At the same time, it outperforms the existing codecs in terms of the mAP for object detection and segmentation.

show abstract

“…However, its traditional HEVC framework still process the image at pixel level and lacks the interpretation ability. A detection-driven image compression with semantically structured bit-stream is proposed in [19], where each part of the bitstream represents a specific object. Later on, the work [42] generalizes the semantically structured image coding framework to multiple intelligent tasks.…”

Section: Introductionmentioning

confidence: 99%

Towards Semantic Communications: Deep Learning-Based Image Semantic Coding

Huang¹,

Gao²,

Tao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Semantic communications has received growing interest since it can remarkably reduce the amount of data to be transmitted without missing critical information. Most existing works explore the semantic encoding and transmission for text and apply techniques in Natural Language Processing (NLP) to interpret the meaning of the text. In this paper, we conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level.Firstly, we define the semantic concept of image data that includes the category, spatial arrangement, and visual feature as the representation unit, and propose a convolutional semantic encoder to extract semantic concepts. Secondly, we propose the image reconstruction criterion that evolves from the traditional pixel similarity to semantic similarity and perceptual performance. Thirdly, we design a novel RL-based semantic bit allocation model, whose reward is the increase in rate-semantic-perceptual performance after encoding a certain semantic concept with adaptive quantization level. Thus, the taskrelated information is preserved and reconstructed properly while less important data is discarded.Finally, we propose the Generative Adversarial Nets (GANs) based semantic decoder that fuses both locally and globally features via an attention module. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image, and saves times of bit cost compared to standard codecs and other deep learning-based image codecs.

show abstract

Beyond Coding: Detection-driven Image Compression with Semantically Structured Bit-stream

Cited by 16 publications

References 9 publications

Compressive Domain Deep CNN for Image Classification and Performance Improvement Using Genetic Algorithm-Based Sensing Mask Learning

Compressive Domain Deep CNN for Image Classification and Performance Improvement Using Genetic Algorithm-Based Sensing Mask Learning

A New Image Codec Paradigm for Human and Machine Uses

Towards Semantic Communications: Deep Learning-Based Image Semantic Coding

Contact Info

Product

Resources

About