On the Link Between Subjective Score Prediction and Disagreement of Video Quality Metrics

Tiotsop, Lohic Fotio; Agboma, Florence; Wallendael, Glenn Van; Aldahdooh, Ahmed; Bosse, Sebastian; Janowski, Lucjan; Barkowsky, Marcus; Masala, Enrico

doi:10.1109/access.2021.3127395

Cited by 4 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another question of high interest for the design of more accurate AIOs is how to collect enough reliable subjective raw opinion scores in order to get rid of synthetic labels or transfer learning approaches. This can be related to our work published in [59]. In fact, new recommendations for the design of subjective tests aiming at the training of AIOs needs to be investigated.…”

Section: Discussionmentioning

confidence: 72%

Predicting individual quality ratings of compressed images through deep CNNs-based artificial observers

Tiotsop

Servetti

Barkowsky

et al. 2023

Signal Processing: Image Communication

Self Cite

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 72%

Predicting individual quality ratings of compressed images through deep CNNs-based artificial observers

Tiotsop

Servetti

Barkowsky

et al. 2023

Signal Processing: Image Communication

Self Cite

View full text Add to dashboard Cite

“…Pearson correlation, RMSE, Outlier Ratio, and variations like Spearman correlation fail to acknowledge or accommodate the variation in opinion among subjects. For example, Pinson et al [10] Tiotsop et al [17] adopt a completely different approach of analyzing the differences among multiple objective metrics, without any subjective data. If multiple metrics agree, the commonly indicated ratings are likely to be accurate.…”

Section: Analyzing Objective Metricsmentioning

confidence: 99%

The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods

Pinson

2023

IEEE Trans. on Broadcast.

View full text Add to dashboard Cite

This paper calculates confidence intervals for 89 datasets that use the 5-level Absolute Category Rating (ACR) method to evaluate the quality of speech, video, images, and video with audio. This data allows us to compute the subjective test confidence interval ( S CI ) for 5-level ACR tests. We use a confusion matrix to compare conclusions reached by 88 lab-to-lab comparisons, 22 method-to-method comparisons, and 12 comparisons between expert and naïve subjects. We estimate the differences in conclusions reached by ad hoc evaluations, compared to subjective tests. We recommend using the disagree incidence rate to identify lab-to-lab differences (i.e., the likelihood that significantly different stimulus pairs receive opposing rank order from the two labs). Disagree incidence rates above 0.31% are unusual enough to warrant investigation and disagree incidence rates above 1.0% indicate differences in method, test environment, test implementation, or subject demographics. These incidence rates form the basis for a new statistical method that calculates the confidence interval of a metric ( M CI ). When M CI is used to make decisions, the equivalence to a video-quality test (EVQT) method determines whether a metric acts similarly to a subjective test. When M CI is not used, the metric is likened to a certain number of people in a video-quality test (PVQT). This information will help users make the better decisions when applying quality metrics. The algorithm code is made available for any purpose. Most of the ratings used in this paper come from open datasets.

show abstract

Multiple Image Distortion DNN Modeling Individual Subject Quality Assessment

Fotio Tiotsop,

Servetti,

Pocta

et al. 2024

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

A recent research direction is focused on training Deep Neural Networks (DNNs) to replicate individual subject assessments of media quality. These DNNs are referred to as Artificial Intelligence-based Observers (AIOs). An AIO is designed to simulate, in real-time, the quality ratings of a specific individual, enabling an automatic quality assessment that accounts for subjects characteristics and preferences. Training AIOs is a promising but challenging research area due to the greater noise in individual raw opinion scores compared to the Mean Opinion Score (MOS). Effective learning from noisy labels necessitates the training of complex models on large-scale datasets. Unfortunately, this is challenging for AIOs as the media quality assessment community lacks extensive datasets that include individual opinion scores. To address the complexity of the task, we first created a dataset comprising two million samples, with synthetic labels derived from human annotation. We then trained a customized network for image quality assessment, named Multi-Distortion ResNet50 (MDResNet50), on this dataset. The weights of the MDResNet50 were subsequently utilized to initialize the learning process of each AIO, thereby avoiding the need to train a complex model from scratch on a small-scale dataset with raw individual opinion scores. Computational experiments show that our approach significantly advances the state-of-the-art in the AIO research. In particular: i) we demonstrate through a simulation the ability of AIOs to mimic two well-known behavioral characteristics of a subject, i.e.,bias and inconsistency, when scoring the media quality; ii) we train and release DNN-based AIOs that, compared to the state-of-the-art, exhibit a higher performance with a statistical significance in assessing multiple image distortions; iii) we train AIOs that more accurately mimic the sensitivity of real subjects to noise and color saturation, and also better predict the opinion score distribution compared to the state-of-the-art AIOs.

show abstract

On the Link Between Subjective Score Prediction and Disagreement of Video Quality Metrics

Cited by 4 publications

References 28 publications

Predicting individual quality ratings of compressed images through deep CNNs-based artificial observers

Predicting individual quality ratings of compressed images through deep CNNs-based artificial observers

The Precision and Repeatability of Media Quality Comparisons: Measurements and New Statistical Methods

Multiple Image Distortion DNN Modeling Individual Subject Quality Assessment

Contact Info

Product

Resources

About