Convolutional Neural Network-Based Block Up-Sampling for Intra Frame Coding

Abstract

The past decade has witnessed the huge success of deep learning in well-known artificial intelligence applications such as face recognition, autonomous driving, and large language model like ChatGPT. Recently, the application of deep learning has been extended to a much wider range, with neural networkbased video coding being one of them. Neural network-based video coding can be performed at two different levels: embedding neural network-based (NN-based) coding tools into a classical video compression framework or building the entire compression framework upon neural networks. This paper elaborates some of the recent exploration efforts of JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) in the name of neural network-based video coding (NNVC), falling in the former category. Specifically, this paper discusses two major NNbased video coding technologies, i.e. neural network-based intra prediction and neural network-based in-loop filtering, which have been investigated for several meeting cycles in JVET and finally adopted into the reference software of NNVC. Extensive experiments on top of the NNVC have been conducted to evaluate the effectiveness of the proposed techniques. Compared with VTM-11.0 nnvc 1 , the proposed NN-based coding tools in NNVC-4.0 could achieve {11.94%, 21.86%, 22.59%}, {9.18%, 19.76%, 20.92%}, and {10.63%, 21.56%, 23.02%} BD-rate reductions on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations respectively.

Video Compression Based on Spatio-Temporal Resolution Adaptation

“…In addition, if resolution downsampling is applied, the base QP used to encode the low resolution video is reduced by a default value of 6, following the analysis in [7] and [14], in order to achieve a similar bitrate as would be achieved if no resampling was performed. 1…”

“…In this context, several authors have proposed reducing spatial resolution for low bitrate encoding [3], [4], but lack a reliable adaptation technique. Others have developed prediction models [5], [6] or have introduced the resolution adaptation as one of the rate-distortion optimized modes at a block level (CTU) [7] but apply them for H.264 or intra coding only. Regarding temporal adaptation, a few methods for frame rate selection have been proposed in [8] and [9].…”

Deep Video Precoding

“…Thus, Afonso et al propose a spatio-temporal resolution adaptation where a CNN-based super-resolution model is used to reconstruct full-resolution content [19]. Li et al [20] introduce the block adaptive resolution coding framework for intra frame coding, where each block within a frame is either downscaled or coded at original resolution and then upscaled with a trained CNN at the decoder side. This concept was later extended to include P and B frames as well [21].…”

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

“…The final version of record is available at http://dx.doi.org/10.1109/TCSVT.2019.2954474 AV1 [141] ---3.0 --Note: 1 LL/RL/LH/RH denote low-delay low complexity, random-access low complexity, low-delay high-efficiency, and random-access high-efficiency configurations in early HM test conditions. [190][191][192][193], block up-sampling for intra frame coding [194], intra mode decision [195][196][197][198][199], transform [200], rate control [201,202], in-loop filtering/post-processing [203][204][205], arithmetic coding [206], or decoder-end artifact-removal and quality enhancement [207,208].…”