Study of Saliency in Objective Video Quality Assessment

Proceedings of the 27th ACM International Conference on Multimedia

2019

214

245

Quality assessment of in-the-wild videos is a challenging problem because of the absence of reference videos and shooting distortions. Knowledge of the human visual system can help establish methods for objective quality assessment of in-the-wild videos. In this work, we show two eminent effects of the human visual system, namely, content-dependency and temporal-memory effects, could be used for this purpose. We propose an objective no-reference video quality assessment method by integrating both effects into a deep neural network. For content-dependency, we extract features from a pre-trained image classification neural network for its inherent content-aware property. For temporal-memory effects, long-term dependencies, especially the temporal hysteresis, are integrated into the network with a gated recurrent unit and a subjectivelyinspired temporal pooling layer. To validate the performance of our method, experiments are conducted on three publicly available inthe-wild video quality assessment databases: KoNViD-1k, CVD2014, and LIVE-Qualcomm, respectively. Experimental results demonstrate that our proposed method outperforms five state-of-the-art methods by a large margin, specifically, 12.39%, 15.71%, 15.45%, and 18.09% overall performance improvements over the second-best method VBLIINDS, in terms of SROCC, KROCC, PLCC and RMSE, respectively. Moreover, the ablation study verifies the crucial role of both the content-aware features and the modeling of temporalmemory effects. The PyTorch implementation of our method is released at https://github.com/lidq92/VSFA. KEYWORDSvideo quality assessment; human visual system; content dependency; temporal-memory effects; in-the-wild videos ACM Reference Format:

Section: Related Work 21 Video Quality Assessmentmentioning

confidence: 99%

Quality Assessment of In-the-Wild Videos

Proceedings of the 27th ACM International Conference on Multimedia

2019

214

245

“…Classical VQA methods are grounded on different cues, such as structures (Wang et al 2004b(Wang et al , 2012, motion (Seshadrinathan and Bovik 2010; Manasa and Channappayya 2016), energy (Li et al 2016a), saliency (Zhang and Liu 2017;You et al 2014), gradients (Lu et al 2019), or natural video statistics (NVS) (Mittal et al 2016;Saad et al 2014;Sinno and Bovik 2019b). Besides, some VQA methods focus on the fusion of primary features (Freitas et al 2018;Li et al 2016b).…”

Section: Video Quality Assessmentmentioning

confidence: 99%

Unified Quality Assessment of in-the-Wild Videos with Mixed Datasets Training

2021

Int J Comput Vis

Video quality assessment (VQA) is an important problem in computer vision. The videos in computer vision applications are usually captured in the wild. We focus on automatically assessing the quality of in-the-wild videos, which is a challenging problem due to the absence of reference videos, the complexity of distortions, and the diversity of video contents. Moreover, the video contents and distortions among existing datasets are quite different, which leads to poor performance of datadriven methods in the cross-dataset evaluation setting. To improve the performance of quality assessment models, we borrow intuitions from human perception, specifically, content dependency and temporal-memory effects of human visual system. To face the cross-dataset evaluation challenge, we explore a mixed datasets training strategy for training a single VQA model with multiple datasets. The proposed unified framework explicitly includes three stages: relative quality assessor, nonlinear mapping, and dataset-specific perceptual scale alignment, to jointly predict relative quality, perceptual quality, and subjective quality. Experiments are conducted on four publicly available datasets for VQA in the wild, i.e., LIVE-VQC, LIVE-Qualcomm, KoNViD-1k, and CVD2014. The experimental results verify the effectiveness of the mixed datasets training strategy and prove the superior performance of the unified model in comparison with the state-of-the-art models. For reproducible research, we make the PyTorch implementation of our method available at https://github.com/lidq92/MDTVSFA. Keywords Content dependency • In-the-wild videos • Mixed datasets training • Temporal-memory effect • Video quality assessment Communicated by Mei Chen.

“…The design of the two steps is often designed differently. In the extraction of features, the commonly used assistant theories are saliency [7]- [10] and the human visual system (HVS) [11]- [14]. Support vector regression (SVR) [15] is usually used in the regression process.…”

Section: Introductionmentioning

confidence: 99%

Panoramic Video Quality Assessment Based on Non-Local Spherical CNN

Yang

Liu

et al. 2021

IEEE Trans. Multimedia

Panoramic video and stereoscopic panoramic video are essential carriers of virtual reality content, so it is very crucial to establish their quality assessment models for the standardization of virtual reality industry. However, it is very challenging to evaluate the quality of the panoramic video at present. One reason is that the spatial information of the panoramic video is warped due to the projection process, and the conventional video quality assessment (VQA) method is difficult to deal with this problem. Another reason is that the traditional VQA method is problematic to capture the complex global time information in the panoramic video. In response to the above questions, this paper presents an end-to-end neural network model to evaluate the quality of panoramic video and stereoscopic panoramic video. Compared to other panoramic video quality assessment methods, our proposed method combines spherical convolutional neural networks (CNN) and non-local neural networks, which can effectively extract complex spatiotemporal information of the panoramic video. We evaluate the method in two databases, VRQ-TJU and VR-VQA48. Experiments show the effectiveness of different modules in our method, and our method outperforms state-of-the-art other related methods.