Performance of Four Subjective Video Quality Assessment Protocols and Impact of Different Rating Preprocessing and Analysis Methods

Kumcu, Asli; Bombeke, Klaas; Platiša, Ljiljana; Jovanov, Ljubomir; Looy, Jan Van; Philips, Wilfried

doi:10.1109/jstsp.2016.2638681

Cited by 19 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kumcu et al [12] compared expert and non-expert ratings for two different applications: medical imagery and denoising. Their experts and non-experts produced comparable scores except for the rank ordering of two denoising systems.…”

Section: A Related Workmentioning

confidence: 99%

“…Their experts and non-experts produced comparable scores except for the rank ordering of two denoising systems. Recent analyses of [12] and similar datasets indicate that experts and non-experts agree on the rank ordering of stimuli quality, when statistical equivalence is taken into consideration (revision to [9], pending publication). However, experts may have an increased or decreased sensitivity to certain impairments.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Pérez

Janowski

García³

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where three to four team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.

show abstract

Section: A Related Workmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Pérez

Janowski

García³

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…The idea was first presented in [18] and later in [22]. The subject model further extends the toolkit for subjective data analysis [9,20]. It also helps make existing analyses more precise [8,21].…”

Section: Related Workmentioning

confidence: 99%

Describing Subjective Experiment Consistency by p-Value P--P Plot

Nawała

Janowski

Ćmiel

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct-value P-P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure

show abstract

“…Those models are discussed in more detail further in the paper. User model allows to extend analysis of the subjective experiments like the one described in [1], [12], or make the existing analysis more precise [13], [14]. The user model is modified in [15], [16], [17] which leads to new results, e.g.…”

Section: Related Workmentioning

confidence: 99%

Generalized Score Distribution

Janowski¹,

Ćmiel²,

Rusek³

et al. 2019

Preprint

View full text Add to dashboard Cite

A class of discrete probability distributions contains distributions with limited support, i.e. possible argument values are limited to a set of numbers (typically consecutive). Examples of such data are results from subjective experiments utilizing the Absolute Category Rating (ACR) technique, where possible answersAn interesting subclass of those distributions are distributions limited to two parameters: describing the mean value and the spread of the answers, and having no more than one change in the probability monotonicity. In this paper we propose a general distribution passing those limitations called Generalized Score Distribution (GSD). The proposed GSD covers all spreads of the answers, from very small, given by the Bernoulli distribution, to the maximum given by a Beta Binomial distribution. We also show that GSD correctly describes subjective experiments scores from video quality evaluations with probability of 99.7%. A Google Collaboratory website with implementation of the GSD estimation, simulation, and visualization is provided.

show abstract

Performance of Four Subjective Video Quality Assessment Protocols and Impact of Different Rating Preprocessing and Analysis Methods

Cited by 19 publications

References 29 publications

Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Describing Subjective Experiment Consistency by p-Value P--P Plot

Generalized Score Distribution

Contact Info

Product

Resources

About