Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

Chen, Baoliang; Zhu, Lingyu; Li, Guo; Lu, Fangbo; Fan, Hongfei; Wang, Shiqi

doi:10.1109/tcsvt.2021.3088505

Cited by 64 publications

(22 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we compared the generalization performance of our proposed method with some deep learning NR-IQA models. Actually, authors of [68] have tested the deep learning NR-IQA models such as WaDIQaM [69] and SPAQ [70] on VQA databases (KoNViD-1k, LIVE-Qualcomm, and CVD2014) and concluded that their overall performance was not satisfactory due to temporal information being discarded. Furthermore, this article shows that NR-VQA models such as VSFA and CNN-TLVQM present better generalization performance than those deep learning NR-IQA models.…”

Section: ) Traditional Vqa Databasesmentioning

confidence: 99%

No-Reference Video Quality Assessment Using Distortion Learning and Temporal Attention

Kossi¹,

Coulombe²,

Desrosiers³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

The rapid growth of video consumption and multimedia applications has increased the interest of the academia and industry in building tools that can evaluate perceptual video quality. Since videos might be distorted when they are captured or transmitted, it is imperative to develop reliable methods for no-reference video quality assessment (NR-VQA). To date, most NR-VQA models in prior art have been proposed for assessing a specific category of distortion, such as authentic distortions or traditional distortions. Moreover, those developed for both authentic and traditional distortions video databases have so far led to poor performances. This resulted in the reluctance of service providers to adopt multiple NR-VQA approaches, as they prefer a single algorithm capable of accurately estimating video quality in all situations. Furthermore, many existing NR-VQA methods are computationally complex and therefore impractical for various real-life applications. In this paper, we propose a novel deep learning method for NR-VQA based on multi-task learning where the distortion of individual frames in a video and the overall quality of the video are predicted by a single neural network. This enables to train the network with a greater amount and variety of data, thereby improving its performance in testing. Additionally, our method leverages temporal attention to select the frames of a video sequence which contribute the most to its perceived quality. The proposed algorithm is evaluated on five publicly-available video quality assessment (VQA) databases containing traditional and authentic distortions. Results show that our method outperforms the state-of-theart on traditional distortion databases such as LIVE VQA and CSIQ video, while also delivering competitive performance on databases containing authentic distortions such as KoNViD-1k, LIVE-Qualcomm and CVD2014.

show abstract

Section: ) Traditional Vqa Databasesmentioning

confidence: 99%

No-Reference Video Quality Assessment Using Distortion Learning and Temporal Attention

Kossi¹,

Coulombe²,

Desrosiers³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Li [12] et al extracted features from the pre-trained image classification neural network ResNet50 to obtain perception characteristics. Chen et al [13] adopted VGG-16 network to learn the frame-level features of videos, and then obtained the Gaussian distributed features through adversarial learning.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, although deep learning can solve this problem, due to the lack of large-scale compressed video quality evaluation databases, the development of datadriven compressed video quality evaluation methods is still not perfect. The content perception features extracted by several methods of transfer learning [12], [13] are all in the spatial domain, which are obviously not enough for videos. To solve the above problems, it is necessary to construct a large compressed video quality evaluation database and propose a data-driven method to learn their spatiotemporal features.…”

Section: Related Workmentioning

confidence: 99%

“…(1) Performance comparison on different databases To verify the performance of the proposed STFEE, it is evaluated on the CVSAR, LIVE, CSIQ and WaterlooIVC4KV databases. The method of STFEE is also compared with typical video quality metrics including PSNR, SSIM, MS-SSIM [36], 3D-PSD [37], SpEED-VQA [7], NIQE [38], VIIDEO [23], TLVQM [39], VSFA [12], MDTVSFA [40], GSTVQA [13] and Shen's algorithm [41] to show its performance. We obtain the source codes of these metrics from the author's public websites.…”

Section: B Performance Comparisonmentioning

confidence: 99%

See 1 more Smart Citation

Deep Quality Assessment of Compressed Videos: A Subjective and Objective Study

Lin¹,

Wang²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the video coding process, the perceived quality of a compressed video is evaluated by full-reference quality evaluation metrics. However, it is difficult to obtain reference videos with perfect quality. To solve this problem, it is critical to design no-reference compressed video quality assessment algorithms, which assists in measuring the quality of experience on the server side and resource allocation on the network side. Convolutional Neural Network (CNN) has shown its advantage in Video Quality Assessment (VQA) with promising successes in recent years. A large-scale quality database is very important for learning accurate and powerful compressed video quality metrics. In this work, a semi-automatic labeling method is adopted to build a large-scale compressed video quality database, which allows us to label a large number of compressed videos with manageable human workload. The resulting Compressed Video quality database with Semi-Automatic Ratings (CVSAR), so far the largest of compressed video quality database. We train a no-reference compressed video quality assessment model with a 3D CNN for SpatioTemporal Feature Extraction and Evaluation (STFEE). Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics and achieves promising generalization performance in cross-database tests. The CVSAR database and STFEE model will be made publicly available to facilitate reproducible research.

show abstract

“…Ying et al [10] demonstrated that features expressive of both combining local patch quality and global video quality can yield significant performance gains. GSTVQA [16] employs a pyramid temporal aggregation of short-term and long-term memory effects to achieve efficient quality prediction. Li et al [17] employs transfer learning on IQA databases, along with a list-wise ranking loss objective to achieve competitive performance.…”

Section: A Nr-vqa Modelsmentioning

confidence: 99%

CONVIQT: Contrastive Video Quality Estimator

Madhusudana¹,

Birkbeck²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Distortion type identification and degradation level determination is employed as an auxiliary task to train a deep learning model containing a deep Convolutional Neural Network (CNN) that extracts spatial features, as well as a recurrent unit that captures temporal information. The model is trained using a contrastive loss and we therefore refer to this training framework and resulting model as CONtrastive VIdeo Quality EstimaTor (CONVIQT). During testing, the weights of the trained model are frozen, and a linear regressor maps the learned features to quality scores in a no-reference (NR) setting. We conduct comprehensive evaluations of the proposed model on multiple VQA databases by analyzing the correlations between model predictions and groundtruth quality ratings, and achieve competitive performance when compared to state-of-the-art NR-VQA models, even though it is not trained on those databases. Our ablation experiments demonstrate that the learned representations are highly robust and generalize well across synthetic and realistic distortions. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning. The implementations used in this work have been made available at https://github.com/pavancm/CONVIQT.

show abstract

Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

Cited by 64 publications

References 59 publications

No-Reference Video Quality Assessment Using Distortion Learning and Temporal Attention

No-Reference Video Quality Assessment Using Distortion Learning and Temporal Attention

Deep Quality Assessment of Compressed Videos: A Subjective and Objective Study

CONVIQT: Contrastive Video Quality Estimator

Contact Info

Product

Resources

About