Attention-Based Dual-Scale CNN In-Loop Filter for Versatile Video Coding

Abstract

As the upcoming video coding standard, Versatile Video Coding (i.e., VVC) achieves up to 30% Bjøntegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (H.265/HEVC). To eliminate or alleviate different kinds of compression artifacts like blocking, ringing, blurring and contouring effects, three in-loop filters, i.e. de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF), have been involved in VVC. Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in many tasks in image processing. In this work, we design a CNN-based in-loop filter as an integrated single-model solution which is adaptive to almost any scenarios in video coding. An architecture named as ADCNN (i.e., Attention based Dual-scale CNN) with an attention based processing block is proposed to reduce artifacts of I frames and B frames, which take advantage of informative priors such as the quantization parameter (QP) and partitioning information. Different from existing CNN-based filtering methods, which are mainly designed for the luma component and may need to train different models for different QPs, the proposed filter is adapted to different QPs and different frame types, and all the components (i.e., both luma and chroma) are processed simultaneously with feature exchange and fusion between components for information supplementary. Experimental results show that the proposed ADCNN filter can achieve 6.54%, 13.27%, 15.72% BD-rate savings for Y, U, V respectively under the all intra configuration and 2.81%, 7.86%, 8.60% BD-rate savings under the random access configuration. It can be used to replace all the conventional in-loop filters and also outperforms them without increase in encoding time. INDEX TERMSCoding artifacts, convolutional neural network (CNN), in-loop filter, video coding, versatile video coding (VVC).

An efficient post-processing adaptive filtering technique to rectifying the flickering effects

“…Due to a one-dimensional filtering approach, the in-loop processing method enhances the coding efficiency by reducing blocking artifacts amongst adjoining pixels or frames but is unable to process corner outliers. To alleviate blocking artifacts different post-processing approaches such as frequency domain analysis , Projection Onto Convex Sets (POCS) [9][10][11][12][13], waveletbased techniques [8,[20][21][22][23][24][25][26][27][28][29][30], estimation theory [5,[9][10][11][12][13], and filtering approach [11][12][13][14][15] has been proposed in last few decades. The most common method is to apply a low-pass filter across block boundaries to remove artifacts.…”

Enhancing VVC Through Cnn-Based Post-Processing

“…The performance of the proposed method for all tested sequences is summarised in Table I, where original VTM 4.01 is used as a benchmark 2 . It is noted that the overall bit rate saving according to PSNR (for luma components only) is 3.76% for low QP range (22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37), which is higher than those for JVET-N0254 and JVET-O0059, and this improvement is evident for all tested video classes (resolutions). When VMAF is employed for quality assessment, the average BD-rates are similar to those for PSNR, at -4.85% for high QP range and -3.40% low QP range.…”

“…Equally, such approaches can be integrated into the encoder as in-loop filters, allowing the frames with enhanced visual quality to also be used as a reference for encoding neighbouring frames when inter prediction is enabled. For the case of VVC, CNN-based post-processing and in-loop filtering approaches have been proposed [25][26][27], which offer bit rate savings for All Intra mode [28]. However the coding gains for the more commonly used Random Access (RA) configuration [28], are relatively low, and the trade-offs between network complexity and overall performance are often not justified.…”

Efficient Adaptation of Neural Network Filter for Video Compression

“…artifacts. CNN-based methods were proposed to replace the conventional filters [8,16] or integrated to be used along with the traditional filtering chain [12].…”