The Method of Increasing the Accuracy of Mean Opinion Score Estimation in Subjective Quality Evaluation

Ostaszewska, A.; Żebrowska-Łucyk, S.

doi:10.1007/978-3-642-15687-8_16

Cited by 6 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A new method of data filtration is presented in [11]. The method proposes the use, for subjects' scores, of Mandel's k and h statistics that were developed for the comparison of inter-laboratory experiments [12], considering "the subject as a laboratory."…”

Section: A Reliability Of Subjectsmentioning

confidence: 99%

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

Pezzulli

Martini

Barman

2021

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion scores using a 5-point scale. The subjects' mean opinion score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample, we may assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper, we propose an approach based on generalized linear models (GLMs) for estimation of the population average QoE. The motivating dataset is composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression ratios. The approach recognizes the multinomial nature of the data and allows for correlation between scores of the same subject. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, particularly for higher bitrates, for which the model estimates present more coherent behavior. Similar convincing results are found on a second dataset, showing the validity of the approach.

show abstract

Section: A Reliability Of Subjectsmentioning

confidence: 99%