Prediction-Aware Quality Enhancement of VVC Using CNN

Abstract

This paper presents a framework for Convolutional Neural Network (CNN)-based quality enhancement task, by taking advantage of coding information in the compressed video signal. The motivation is that normative decisions made by the encoder can significantly impact the type and strength of artifacts in the decoded images. In this paper, the main focus has been put on decisions defining the prediction signal in intra and inter frames. This information has been used in the training phase as well as input to help the process of learning artifacts that are specific to each coding type. Furthermore, to retain a low memory requirement for the proposed method, one model is used for all Quantization Parameters (QPs) with a QP-map, which is also shared between luma and chroma components. In addition to the Post Processing (PP) approach, the In-Loop Filtering (ILF) codec integration has also been considered, where the characteristics of the Group of Pictures (GoP) are taken into account to boost the performance. The proposed CNN-based Quality Enhancement (QE) framework has been implemented on top of the Versatile Video Coding (VVC) Test Model (VTM-10). Experiments show that the prediction-aware aspect of the proposed method improves the coding efficiency gain of the default CNN-based QE method by 1.52%, in terms of BD-BR, at the same network complexity compared to the default CNN-based QE filter.

A CNN-Based Prediction-Aware Quality Enhancement Framework for VVC

“…• QP-specific training: dedicating one model for each QP or a range of QPs [8,66]. • QP-map training: providing QP as an input to the network [12,27,33,50,57].…”

“…The best IPM, minimizing the R-D cost of a block, is not necessarily the IPM that represents the block texture most accurately [66]. An example of such a situation is presented in Fig.…”

“…2 proves that encoder decisions can have major impact on the QE task and its performance. Particularly, for intra blocks, we assume that the selected IPM for a block carries important information and shall be included in the training of the proposed CNN-based QE method [66]. Therefore, an intra prediction frame is constructed by concatenating the intra prediction signals in the block-level.…”

Model Selection CNN-based VVC QualityEnhancement

“…The selection of an IPM for a block is performed by optimizing the rate-distortion (R-D) cost. Due to the complex R-D optimization process, the generated intra prediction can significantly affect the distortion pattern of the reconstructed block [8]. Thus, bewaring the QE networks of the prediction signal can potentially help it learn correlation between the prediction signal and the distortion patterns.…”

“…In the second group, multiframe methods [3], [4], the temporal aspect of the video is also taken into account for QE task. In addition, some methods [5]- [8] in these two groups, exploit coding information, extracted from bitstream, in order to further improve the quality. Such coding information (e.g.…”