On the Accuracy of Open Video Quality Metrics for Local Decision in AV1 Video Codec

Pastor, Andréas; Krasula, Lukáš; Zhu, Xiaoqing; Li, Zhi; Callet, Patrick Le

doi:10.1109/icip46576.2022.9897469

Cited by 6 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first dataset is from an "in-lab" experiment on the quality of video patches, and the second one, named "crowdsourced" is its reproduction on Prolific 1 , a crowdsourcing platform. These two datasets are subsets of the dataset presented in [25] on 12 out of the 20 contents. The task of participants, see visual example in figure 1, was to provide a preference on a quadruplet (a,b,c,d): where do you perceive a greater difference between a first pair of video patches and a second pair of video patches?…”

Section: Subjective Experiments Datamentioning

confidence: 99%

Recovering Quality Scores in Noisy Pairwise Subjective Experiments Using Negative Log-Likelihood

Pastor,

Krasula,

Zhu

et al. 2023

2023 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

To gather larger datasets to train data-angry deep learning quality assessment models, crowdsourcing has become essential to recruit participants. These participants are asked their opinion by directly rating stimuli, e.g., using single or double stimulus methodologies, or indirectly by ranking stimuli or comparing distances as in the Maximum Likelihood Difference Scaling method. In crowdsourcing, participants' behaviors and environmental distractions are not controlled. So, the researcher must pay attention to the answers' reliability. Cleaning methods exist for direct annotation subjective methodologies. However, solutions for indirect annotation methods are limited. In this work, we propose a method based on the negative log-likelihood to detect spammers among participants from their answers. To demonstrate its use, we applied it in a quadruplet preference-based scenario. The proposed method requires low computation and can be integrated into active-sampling strategies, where annotations available per comparison are small. We demonstrate that our method is robust to various spammer behaviors and accurate by removing only spammers. It helps reduce the gap between data collected in in-lab conditions (i.e., no spammer) and through crowdsourcing: our method reduces estimated uncertainties around data-points by 50%, and RMSE between estimations from an in-lab experiment and the same experiment performed in crowdsourcing by 1.8.

show abstract

Section: Subjective Experiments Datamentioning

confidence: 99%

Recovering Quality Scores in Noisy Pairwise Subjective Experiments Using Negative Log-Likelihood

Pastor,

Krasula,

Zhu

et al. 2023

2023 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The tube-contents in our subjective dataset are selected out of 100K following the clustering approach suggested in [7] for content selection. This clustering approach relies on the responses of various Full-Reference quality metrics on the degraded tubes.…”

Section: Subjective Datasetmentioning

confidence: 99%

“…The work in [7] proposes a benchmark of image and video quality metrics on small video tubes. These tubes were subjectively annotated with how humans perceived the distortion introduced by AV1 encoding.…”

Section: Introductionmentioning

confidence: 99%

Predicting local distortions introduced by AV1 using Deep Features

Pastor,

Krasula,

Zhu

et al. 2023

2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)

Self Cite

View full text Add to dashboard Cite

Semantics extracted by filters in deep learning networks correlate well with how human eyes perceive distortions. These methods (e.g., LPIPS, PieAPP, etc.) rely on the relative difference in activation between feature maps in pairs of references and distorted patches. However, Deep Feature extraction can be expensive to compute as a difference of latent code between reference and distorted frames. Therefore, it is challenging to integrate them into the decision process of modern video codecs like AV1, making thousands of encoding trials during exhaustive Rate-Distortion Optimization (RDO) searches. In this study, we present a method using deep features to predict the distortion perceived locally by human eyes in AV1-encoded videos. The prediction relies on Deep Features extracted from the reference frame only to weigh the Mean Squared Error (MSE) introduced during encoding. This approach will make integration into video codecs easier as a pre-processing step before starting encoding. We show the superiority of the proposed metric against other Reference-Only metrics on a dataset of local distortions in videos. We achieve comparable performance as state-of-the-art Full-Reference video quality metrics.

show abstract

“…In [30], we proposed a modified version of the MLDS methodology to perform inter-content scaling. In [29], we applied the proposed method to start to collect data on small video patches.…”

Section: Research Questionsmentioning

confidence: 99%

Perception of video quality at a local spatio-temporal horizon

Pastor

Callet

2022

Proceedings of the 13th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

This paper contains the research proposal of Andréas Pastor that was presented at the MMSys 2022 doctoral symposium. Encoding video for streaming on Internet has become a major topic to reduce the consumption of bandwidth and latency. At the same time, the human perception of distortions has been explored in multiple research projects, especially for distortions generated by Coder-DECoder (CODEC) algorithms. These algorithms operate in a rate-distortion optimization paradigm to efficiently compress video content. This optimization can be driven by metrics that are most of the time not based on the human perception, and more importantly, not tuned to reflect the local perception of distortions by human eyes.In this doctoral study, we proposed to work on the perception of localized distortion at a small temporal and spatial horizon. We present here the fundamental research questions and challenges in the domain with a focus on methods to collect perceptual judgments in subjective studies and metrics that can help us to derive an estimate of the perception of distortions by humans.

show abstract

On the Accuracy of Open Video Quality Metrics for Local Decision in AV1 Video Codec

Cited by 6 publications

References 18 publications

Recovering Quality Scores in Noisy Pairwise Subjective Experiments Using Negative Log-Likelihood

Recovering Quality Scores in Noisy Pairwise Subjective Experiments Using Negative Log-Likelihood

Predicting local distortions introduced by AV1 using Deep Features

Perception of video quality at a local spatio-temporal horizon

Contact Info

Product

Resources

About