Enhanced Bi-Prediction With Convolutional Neural Network for High-Efficiency Video Coding

Zhao, Zhimin; Wang, Shiqi; Zhang, Xinfeng; Ma, Siwei; Yang, Jiansheng

doi:10.1109/tcsvt.2018.2876399

Cited by 64 publications

(29 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, Zhang et al [23] proposed a half-pel interpolation filter based on a superresolution network [35], while Yan et al [24] proposed a different CNN-based interpolation filter. Other studies that have looked at the use of neural networks in motion compensation include [25]- [27]. Huo et al [25] proposed a CNN-based motion compensation refinement algorithm.…”

Section: A Prior Workmentioning

confidence: 99%

“…The approach is less suitable for larger blocks, but it is applicable to both uni-and bi-prediction. In [26], [27], Zhao et al suggested CNN-based bi-directional motion compensation. Conventional bi-prediction averages two predictions unless additional weights are transmitted, while the proposed CNN combines two predictions adaptively, based on the content, and produces more accurate predicted blocks.…”

Section: A Prior Workmentioning

confidence: 99%

“…However, most of the coding gain in video compression comes from inter prediction. Recent works on inter prediction using DNNs include interpolation filters [23], [24], enhanced motion compensation [25]- [27], and texture prediction, which directly generates the predicted pixel values using reference samples/frames as inputs to the trained network [28] without transferring motion information. In this paper, we propose a DNN-based frame prediction architecture that is able to support both uni-and bi-directional prediction.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Deep Frame Prediction for Video Coding

Choi

Bajić

2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

We propose a novel frame prediction method using a deep neural network (DNN), with the goal of improving video coding efficiency. The proposed DNN makes use of decoded frames, at both encoder and decoder, to predict textures of the current coding block. Unlike conventional inter-prediction, the proposed method does not require any motion information to be transferred between the encoder and the decoder. Still, both uni-directional and bi-directional prediction are possible using the proposed DNN, which is enabled by the use of the temporal index channel, in addition to color channels. In this study, we developed a jointly trained DNN for both uniand bi-directional prediction, as well as separate networks for uni-and bi-directional prediction, and compared the efficacy of both approaches. The proposed DNNs were compared with the conventional motion-compensated prediction in the latest video coding standard, HEVC, in terms of BD-Bitrate. The experiments show that the proposed joint DNN (for both uniand bi-directional prediction) reduces the luminance bitrate by about 4.4%, 2.4%, and 2.3% in the Low delay P, Low delay, and Random access configurations, respectively. In addition, using the separately trained DNNs brings further bit savings of about 0.3%-0.5%.

show abstract

Section: A Prior Workmentioning

confidence: 99%

Section: A Prior Workmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Deep Frame Prediction for Video Coding

Choi

Bajić

2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Current inter-prediction techniques using CNN VPNs aim to generate synthesized reference blocks using the previously coded signals [5]- [8], [28]. Zhao et al propose a bidirectional (B) motion prediction technique using CNN VPN to infer a prediction block [6]. Conventional B-prediction signal is enhanced by the network to provide further temporal correlation.…”

Section: B Cnn-based Inter Coding Techniquesmentioning

confidence: 99%

Deep Video Prediction Network-Based Inter-Frame Coding in HEVC

et al. 2020

View full text Add to dashboard Cite

In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides −2.9% and −5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.

show abstract

“…In [188], a group variational transformation convolutional neural network (GVTCNN) were designed to improve the fractional interpolation performance of the luma component in motion compensation, which infers samples at different sub-pixel positions from the input integer-position sample. In [189], a CNNbased bi-prediction model was presented, in which the predictive signal can be automatically inferred in an end-to-end manner, and the non-linear mapping leads to better fusion in the bi-prediction process than conventional block-wise translational motion mapping. Chen proposed the concept of VoxelCNN [210], which includes motion extension and hybrid prediction networks, can model spatio-temporal coherence to effectively perform predictive coding inside the learning network.…”

Section: Finer Precision Motion Estimation and Compensationmentioning

confidence: 99%

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

Zhang

Fan

et al. 2020

IEEE Trans. Circuits Syst. Video Technol.

Self Cite

View full text Add to dashboard Cite

High Efficiency Video Coding (HEVC) has doubled the video compression ratio with equivalent subjective quality as compared to its predecessor H.264/AVC. The significant coding efficiency improvement is attributed to many new techniques. Interframe coding is one of the most powerful yet complicated techniques therein and has posed high computational burden thus main obstacle in HEVC-based real-time applications. Recently, plenty of research has been done to optimize the inter-frame coding, either to reduce the complexity for real-time applications, or to further enhance the encoding efficiency. In this paper, we provide a comprehensive review of the state-of-the-art techniques for HEVC inter-frame coding from three aspects, namely fast inter coding solutions, implementation on different hardware platforms as well as advanced inter coding techniques. More specifically, different algorithms in each aspect are further subdivided into sub-categories and compared in terms of pros, cons, coding efficiency and coding complexity. To the best of our knowledge, this is the first such comprehensive review of the recent advances of the inter-frame coding for HEVC and hopefully it would help the improvement, implementation and applications of HEVC as well as the ongoing development of the next generation video coding standard.

show abstract

Enhanced Bi-Prediction With Convolutional Neural Network for High-Efficiency Video Coding

Cited by 64 publications

References 38 publications

Deep Frame Prediction for Video Coding

Deep Frame Prediction for Video Coding

Deep Video Prediction Network-Based Inter-Frame Coding in HEVC

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

Contact Info

Product

Resources

About