Adaptive CU Split Decision Based on Deep Learning and Multifeature Fusion for H.266/VVC

Abstract

With the development of technology, the hardware requirement and expectations of user for visual enjoyment are getting higher and higher. The multitype tree (MTT) architecture is proposed by the Joint Video Experts Team (JVET). Therefore, it is necessary to determine not only coding unit (CU) depth but also its split mode in the H.266/Versatile Video Coding (H.266/VVC). Although H.266/VVC achieves significant coding performance on the basis of H.265/High Efficiency Video Coding (H.265/HEVC), it causes significantly coding complexity and increases coding time, where the most time-consuming part is traversal calculation rate-distortion (RD) of CU. To solve these problems, this paper proposes an adaptive CU split decision method based on deep learning and multifeature fusion. Firstly, we develop a texture classification model based on threshold to recognize complex and homogeneous CU. Secondly, if the complex CUs belong to edge CU, a Convolutional Neural Network (CNN) structure based on multifeature fusion is utilized to classify CU. Otherwise, an adaptive CNN structure is used to classify CUs. Finally, the division of CU is determined by the trained network and the parameters of CU. When the complex CUs are split, the above two CNN schemes can successfully process the training samples and terminate the rate-distortion optimization (RDO) calculation for some CUs. The experimental results indicate that the proposed method reduces the computational complexity and saves 39.39% encoding time, thereby achieving fast encoding in H.266/VVC.

Fast Depth Intra Coding Based on Depth Edge Classification Network in 3D-HEVC

“…As a new research method in recent years, many scholars have applied CNN to the fast coding [25]- [33]. Such a solution was applied in different video coding domains, like the previous generation video coding standard HEVC [25], [26], the latest generation video coding standard VVC [27], [28], and the extended video coding standard 3D-HEVC [29]- [33]. For example, to reduce the HEVC complexity at both intra-and inter-modes, Xu et al [25] proposed a deep learning approach to predict the CU partition based on CNN and long-and short-term memory (LSTM) network.…”

Medical Image Compression Method Using Lightweight Multi-Layer Perceptron for Mobile Healthcare Applications

“…Lin et al [34] proposed a CNN-based intra mode decision method with two convolutional layers and a fully connected layer. Zhao et al [35] developed an adaptive CU split decision method with the deep learning and multi-featured fusion. Zaki et al [36] also proposed a CtuNet framework to support CTU partitioning using deep learning techniques.…”

Fast CU Decision-Making Algorithm Based on DenseNet Network for VVC

“…Due to the irregular shape and size of CUs, some methods based on CNN are used on these CUs. [23], [24] use an adaptive pooling layer to solve this problem. The adaptive pooling layer can compress feature maps of any size into a fixed size.…”