Regularized Maximum Likelihood Estimation of the Subjective Quality from Noisy Individual Ratings

Tiotsop, Lohic Fotio; Servetti, Antonio; Barkowsky, Marcus; Masala, Enrico

doi:10.1109/qomex55416.2022.9900903

Cited by 5 publications

(15 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This yields an approach with a stronger theoretical foundation. This work significantly extends our previous one [13] in which we introduced only the RMLE approach. Here, we extend this previous work by better motivating the RMLE approach and by proposing a novel subject scoring model.…”

Section: Introductionsupporting

confidence: 57%

“…Theorem 1. Under the assumption in Eq (13) and assuming that the random variables θ j ikf f ∈ F are independent, as the number of IFs tends to infinity, i.e., |F| → +∞, the probability that subject j chooses the opinion score k when rating the stimulus i is:…”

Section: B Deriving the Proposed Subject Scoring Modelmentioning

confidence: 99%

“…where for the equality in Eq ( 18) and ( 19), we exploited the independence of the random variables θ j ikf . To obtain (20), we exploit the assumption in Eq (13).…”

Section: B Deriving the Proposed Subject Scoring Modelmentioning

confidence: 99%

“…Following the approach of [11], [12], [35], we benchmarked the performance of our proposed quality recovery approach on real datasets by showing that its estimated subjective quality suffers lower uncertainty than that estimated by the SUREAL software. As in the aforementioned previous papers, here, the level of uncertainty in the estimated subjective quality is measured by the size of the CI.…”

Section: E Assessing the Uncertainty On The Estimated Subjective Qualitymentioning

confidence: 99%

See 3 more Smart Citations

Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

Tiotsop,

Servetti,

Barkowsky

et al. 2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Several approaches have been proposed to estimate quality in subjective experiments while highlighting peculiar subject behaviors. However, there is some room for improvement in existing approaches, both in terms of robustness to noise and the ability to accurately indicate several peculiar subject behaviors in subjective experiments. This work advances the state-of-the-art in three main directions: i) A new approach to estimate the subjective quality from noisy ratings is proposed and is shown to be more robust to noise than are four state-ofthe-art approaches; ii) a novel subject scoring model is proposed that makes it possible to highlight several peculiar behaviors typically observed in subjective experiments; and iii) our proposed probabilistic subject scoring model results from the proof of a theorem, whereas in previous approaches a probabilistic scoring model is assumed a priori. This represents an important first step toward models supported by a stronger theoretical foundation. Numerical experiments conducted on several datasets highlight the effectiveness of our proposal.

show abstract

Section: Introductionsupporting

confidence: 57%

Section: B Deriving the Proposed Subject Scoring Modelmentioning

confidence: 99%

“…where for the equality in Eq ( 18) and ( 19), we exploited the independence of the random variables θ j ikf . To obtain (20), we exploit the assumption in Eq (13).…”

Section: B Deriving the Proposed Subject Scoring Modelmentioning

confidence: 99%

Section: E Assessing the Uncertainty On The Estimated Subjective Qualitymentioning

confidence: 99%

See 2 more Smart Citations

Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

Tiotsop,

Servetti,

Barkowsky

et al. 2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…Despite its popularity, the MOS is known to be particularly sensitive to outlier opinion scores [5]. As a consequence, more sophisticated approaches that exploit parametric statistical models have recently been proposed for the subjective media quality recovery problem [3], [6]- [8].…”

Section: Introductionmentioning

confidence: 99%

Subjective Media Quality Recovery From Noisy Raw Opinion Scores: A Non-Parametric Perspective

Altieri,

Tiotsop,

Valenzise

2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

This paper focuses on the challenge of accurately estimating the subjective quality of multimedia content from noisy opinion scores gathered from end-users. State-of-the-art methods rely on parametric statistical models to capture the subject's scoring behavior and recover quality estimates. However, these approaches have limitations, as they often require restrictive assumptions to achieve numerical stability during parameter estimation, leading to a lack of robustness when the modeling hypotheses do not fit the data. To overcome these limitations, we propose a paradigm shift towards non-parametric statistical methods. Specifically, we introduce a threefold contribution: i) in contrast to the prevailing approach in subjective quality recovery assuming a parametric score distribution, we propose a non parametric approach that guarantees greater accuracy by measuring reliability per subject and per stimulus, overcoming the limits of existing approaches that measure only per subject reliability; ii) we propose ESQR, a non-parametric algorithm for subjective quality recovery, demonstrating experimentally that it has higher robustness to noise compared to numerous state-of-the-art algorithms, thanks to the weaker assumptions made on data compared to parametric approaches; iii) the proposed approach is theoretically grounded, i.e., we define a nonparametric statistic and prove mathematically that it provides a measure of score reliability. The code to run ESQR and reproduce the results in this paper is made freely available at: http://media.polito.it/ESQR.

show abstract

Comparing Simulated and Real Conversations for QoE Assessments: Insights from ARKit-Based Facial Configuration Analyses

Spang¹,

Wardah²,

Schmitt

et al. 2023

2023 15th International Conference on Quality of Multimedia Experience (QoMEX)

View full text Add to dashboard Cite

Regularized Maximum Likelihood Estimation of the Subjective Quality from Noisy Individual Ratings

Cited by 5 publications

References 3 publications

Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

Subjective Media Quality Recovery From Noisy Raw Opinion Scores: A Non-Parametric Perspective

Comparing Simulated and Real Conversations for QoE Assessments: Insights from ARKit-Based Facial Configuration Analyses

Contact Info

Product

Resources

About