Stereoscopic video quality assessment based on 3D convolutional neural networks

Abstract

Keywords:3D convolutional neural networks Stereoscopic video quality assessment Quality score fusion a b s t r a c tThe research of stereoscopic video quality assessment (SVQA) plays an important role for promoting the development of stereoscopic video system. Existing SVQA metrics rely on hand-crafted features, which is inaccurate and time-consuming because of the diversity and complexity of stereoscopic video distortion. This paper introduces a 3D convolutional neural networks (CNN) based SVQA framework that can model not only local spatio-temporal information but also global temporal information with cubic difference video patches as input. First, instead of using hand-crafted features, we design a 3D CNN architecture to automatically and effectively capture local spatio-temporal features. Then we employ a quality score fusion strategy considering global temporal clues to obtain final video-level predicted score. Extensive experiments conducted on two public stereoscopic video quality datasets show that the proposed method correlates highly with human perception and outperforms state-of-the-art methods by a large margin. We also show that our 3D CNN features have more desirable property for SVQA than hand-crafted features in previous methods, and our 3D CNN features together with support vector regression (SVR) can further boost the performance. In addition, with no complex preprocessing and GPU acceleration, our proposed method is demonstrated computationally efficient and easy to use. (W. Lu).Unfortunately, considering that reference video is unavailable in most practical applications, only the NR methods have potential to satisfy the actual requirement. As a result, our work focuses on more appealing and challenging NR methods, and tries to propose a new general-purpose NR framework for SVQA.Considering the development of NR metrics [1,2] , we can find that most of them have similar frameworks, which can be generally divided into two steps: (1) extracts features that can reflect visual quality based on relatively perceptual models; (2) maps the obtained feature vectors to subjective quality scores by learning regression models. But in real-world scenarios, because of the diversity and complexity of video distortions, it is very difficult to identify what features are sensitive and robust to all sorts of distortions. Therefore, simply utilizing a group of artificially designed features to represent video quality results in inaccurate assessments and high computational costs.Recently, it is a well-known fact that deep learning models, especially convolutional neural networks (CNN), have achieved great success in many challenge computer vision tasks, such as image classification [3,4] , object detection [5,6] , video classification [7,8] and video action recognition [9,10] . CNN is a biologically inspired architecture consisting of a stack of convolutional layers

A Preliminary Experimental Analysis of In-Pipe Image Transmission Based on Visible Light Relay Communication

“…Furthermore, for DIFRT, the relationship between the BER and image quality has also been compared and analyzed. In particular, to guarantee the accuracy of experiments, the interference from ambient light has been strictly avoided [38,39,40].…”

Blind quality assessment for 3D synthesised video with binocular asymmetric distortion

“…Wang et al [30] proposed a VQA evaluator for asymmetric compression distortion video based on different weights of left and right viewpoints. Yang et al [31] proposed an SVQA method, which extracts local features in spatial and temporal domains by a 3D convolutional neural network (CNN).…”

Modern trends on quality of experience assessment and future work

“…Also, Ding et al [107] considered various HVS factors such as saliency and multiscale disparity map, which produced better performance than previous works. Secondly, we compared the performance of existing S3D VQA models: Feng [111], PHVS-3D [97], SFD [98], 3D-STS [108], MNSVQM [113], BSVQE [112], and Yang [114]. The IRCCYN 3D video quality database [97] and the Qi stereoscopic video quality database [98] were used for performance comparison.…”

“…Secondly, we compared the performance of existing S3D VQA models: Feng [111], PHVS-3D [97], SFD [98], 3D-STS [108], MNSVQM [113], BSVQE [112], and Yang [114]. The IRCCYN 3D video quality database [97] and the Qi stereoscopic video quality database [98] were used for performance comparison.…”

“…As can be seen in Table 12, the performance of S3D VQA is better when they are trained on the QI database. Because the QI database has fewer distortion types and more training videos, the VQA than the other methods while learning the spatial and temporal characteristics of videos more successfully [114].…”