Deep Perceptual Preprocessing for Video Coding

Chadha, Aaron; Andreopoulos, Yiannis

doi:10.1109/cvpr46437.2021.01461

Cited by 22 publications

(20 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, several learning-based preprocessing methods have been proposed [12,18,33,34]. Chadha et al [12] proposed a rate-aware perceptual preprocessing module for video coding.…”

Section: Preprocessingmentioning

confidence: 99%

See 1 more Smart Citation

Preprocessing Enhanced Image Compression for Machine Vision

Guo¹,

Ge²,

Tianxiong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, more and more images are compressed and sent to the back-end devices for the machine analysis tasks (e.g., object detection) instead of being purely watched by humans. However, most traditional or learned image codecs are designed to minimize the distortion of the human visual system without considering the increased demand from machine vision systems. In this work, we propose a preprocessing enhanced image compression method for machine vision tasks to address this challenge. Instead of relying on the learned image codecs for endto-end optimization, our framework is built upon the traditional non-differential codecs, which means it is standard compatible and can be easily deployed in practical applications. Specifically, we propose a neural preprocessing module before the encoder to maintain the useful semantic information for the downstream tasks and suppress the irrelevant information for bitrate saving. Furthermore, our neural preprocessing module is quantization adaptive and can be used in different compression ratios. More importantly, to jointly optimize the preprocessing module with the downstream machine vision tasks, we introduce the proxy network for the traditional non-differential codecs in the back-propagation stage. We provide extensive experiments by evaluating our compression method for two representative downstream tasks with different backbone networks. Experimental results show our method achieves a better trade-off between the coding bitrate and the performance of the downstream machine vision tasks by saving about 20% bitrate.Preprint. Under review.

show abstract

“…In recent years, several learning-based preprocessing methods have been proposed [12,18,33,34]. Chadha et al [12] proposed a rate-aware perceptual preprocessing module for video coding.…”

Section: Preprocessingmentioning

confidence: 99%

“…In recent years, several learning-based preprocessing methods have been proposed [12,18,33,34]. Chadha et al [12] proposed a rate-aware perceptual preprocessing module for video coding. Onur et al [18] proposed neural network based preprocessing and postprocessing modules to improve the compression performance of the traditional codecs.…”

Section: Preprocessingmentioning

confidence: 99%

Preprocessing Enhanced Image Compression for Machine Vision

Guo¹,

Ge²,

Tianxiong³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The results, shown in Table 2, demonstrate that sharpening approaches cannot provide for universal BD-rate gains across different metrics since, unlike the proposed approach, they change the perceptual characteristics of the source. Moreover, while previously-proposed preprocessing [8]…”

Section: Bd Rates (%)mentioning

confidence: 99%

“…offers BDrate gains for SSIM and VMAF, it is outperformed by the present approach. BD-rate averages on the XIPH content for sharpening and two previous deep perceptual preprocessing models[8]. Positive numbers indicate average bitrate increase and more negative numbers indicate more saving versus the corresponding encoder.Finally, on-going work is showing that the obtained savings are also validated by rigorous controlled tests with human viewers.…”

mentioning

confidence: 99%

Toward Generalized Psychovisual Preprocessing For Video Encoding

Chadha¹,

Anam²,

Treder³

et al. 2022

SMPTE Mot. Imag. J

Self Cite

View full text Add to dashboard Cite

Deep perceptual preprocessing has recently emerged as a new way to enable further bitrate savings across several generations of video encoders without breaking standards or requiring any changes in client devices. In this paper, we lay the foundations toward a generalized psychovisual preprocessing framework for video encoding and describe one of its promising instantiations that is practically deployable for video-on-demand, live, gaming and user-generated content. Results using state-of-the-art AVC, HEVC and VVC encoders show that average bitrate (BD-rate) gains of 11% to 17% are obtained over three state-of-the-art reference-based quality metrics (Netflix VMAF, SSIM and Apple AVQT), as well as the recently-proposed non-reference ITU-T p.1204 metric. The proposed framework on CPU is shown to be twice faster than x264 mediumpreset encoding. On GPU hardware, our approach achieves 714fps for 1080p video (below 2ms/frame), thereby enabling its use in very-low latency live video or game streaming applications.

show abstract

“…Despite the pioneer of CNN based blind IQA [2] adopts minmax-pooling, the majority of subsequent works [4][5] [12][13] adopts SAP (include global average pooling) as it has become popular in CNN. [14] adopts a mixture of SAP and ROI Pooling.…”

Section: Poolings and Deep Cnn Based Blind Iqamentioning

confidence: 99%

Spatial Moment Pooling Improves Neural Image Assessment

Xu¹,

Shao

Wang

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In recent years, there has been widespread attention drawn to convolutional neural network (CNN) based blind image quality assessment (IQA). A large number of works start by extracting deep features from CNN. Then, those features are processed through spatial average pooling (SAP) and fully connected layers to predict quality. Inspired by full reference IQA and texture features, in this paper, we extend SAP (1 st moment) into spatial moment pooling (SMP) by incorporating higher order moments (such as variance, skewness). Moreover, we provide learning friendly normalization to circumvent numerical issue when computing gradients of higher moments. Experimental results suggest that simply upgrading SAP to SMP significantly enhances CNN-based blind IQA methods and achieves state of the art performance.

show abstract

Deep Perceptual Preprocessing for Video Coding

Cited by 22 publications

References 37 publications

Preprocessing Enhanced Image Compression for Machine Vision

Preprocessing Enhanced Image Compression for Machine Vision

Toward Generalized Psychovisual Preprocessing For Video Encoding

Spatial Moment Pooling Improves Neural Image Assessment

Contact Info

Product

Resources

About