CNN-based Fast Split Mode Decision Algorithm for Versatile Video Coding (VVC) Inter Prediction

Abstract

Versatile Video Coding (VVC) is the latest video coding standard developed by Joint Video Exploration Team (JVET). In VVC, the quadtree plus multi-type tree (QT+MTT) structure of coding unit (CU) partition is adopted, and its computational complexity is considerably high due to the brute-force search for recursive rate-distortion (RD) optimization. In this paper, we aim to reduce the time complexity of inter-picture prediction mode since the inter prediction accounts for a large portion of the total encoding time. The problem can be defined as classifying the split mode of each CU. To classify the split mode effectively, a novel convolutional neural network (CNN) called multi-level tree (MLT-CNN) architecture is introduced. For boosting classification performance, we utilize additional information including inter-picture information while training the CNN. The overall algorithm including the MLT-CNN inference process is implemented on VVC Test Model (VTM) 11.0. The CUs of size 128×128 can be the inputs of the CNN. The sequences are encoded at the random access (RA) configuration with five QP values {22, 27, 32, 37, 42}. The experimental results show that the proposed algorithm can reduce the computational complexity by 11.53% on average, and 26.14% for the maximum with an average 1.01% of the increase in Bjøntegaard delta bit rate (BDBR). Especially, the proposed method shows higher performance on the sequences of the A and B classes, reducing 9.81%~26.14% of encoding time with 0.95%~3.28% of the BDBR increase.

Light-Weight CNN-Based VVC Inter Partitioning Acceleration

“…More precisely, when we set the threshold Th in Algorithm 1 to 0, we obtain a similar acceleration as the method developed in [8] with a significantly lower loss. Similarly, when the threshold Th equals 0.1, acceleration is tripled and the loss remains the same as in [9].…”

“…In Table 1, the proposed method is compared with the two CNN-based inter partitioning methods in [8] and [9]. For a fair comparison, we have implemented our method in VTM6 and VTM11.…”

“…Various machine learning based methods [6] [7] [8] [9] have been proposed to speed up the RDO search of inter partitioning in VVC. They can be categorized into Random Forest (RF)-based and CNN-based methods.…”

“…In this paper, different models are trained for different CU sizes. Moreover, Yeo et al propose a method in [9], where a variant of the Branch Convolutional Neural Network (B-CNN) [10], called multi-level tree CNN (MLT-CNN), is used to predict the partitioning mode of the CTU. The key feature of this CNN is that its outputs correspond to different decisions at different levels of a split mode tree.…”

Temporal Prediction Model-Based Fast Inter CU Partition for Versatile Video Coding

“…Researchers have proposed many fast CU partition decision methods for reducing the inter coding complexity of VVC [ 30 , 31 , 32 , 33 ]. In [ 30 ], the difference between three frames is extracted to indicate that the CU is a motion block or a static block.…”