Deep Learning-based Synthesized View Quality Enhancement with DIBR Distortion Mask Prediction Using Synthetic Images

Abstract

Recently, deep learning-based image quality enhancement models have been proposed to improve the perceptual quality of distorted synthesized views impaired by compression and Depth Image Based Rendering (DIBR) process in multiview video systems. However, due to the lack of multi-view video plus depth data, the training data for quality enhancement models is small, which limits the performance and progress of these models. Augmenting the training data to enhance the Synthesized View Quality Enhancement (SVQE) models is a feasible solution. In this paper, we suggest a deep learning-based SVQE model using more synthetic Synthesized View Images (SVIs). To simulate the irregular geometric displacement of DIBR distortion, a random irregular polygon-based SVI synthesis method is proposed based on existing massive RGB/RGBD data, and a synthetic synthesized view database is constructed, which includes synthetic SVIs and DIBR distortion masks. Moreover, to further guide the SVQE models to focus more precisely on DIBR distortion, the DIBR distortion mask prediction network which could predict the position and variance of DIBR distortion is embedded into the SVQE models. The experimental results demonstrate that by pretraining on the synthetic SVI database, the performance of the existing SVQE models could be greatly promoted. In addition, by introducing the DIBR distortion mask prediction network, the SVI quality could be further enhanced.