When is the Cleaning of Subjective Data Relevant to Train UGC Video Quality Metrics?

Abstract

Outlier analysis and spammer detection recently gained momentum in order to reduce uncertainty of subjective ratings in image & video quality assessment tasks. The large proportion of unreliable ratings from online crowdsourcing experiments and the need for qualitative and quantitative large-scale studies in the deep-learning ecosystem played a role in this event.We study the effect that data cleaning has on trainable models predicting the visual quality for videos, and present results demonstrating when cleaning is necessary to reach higher efficiency. To this end, we present and analyze a benchmark on clean and noisy User Generated Content (UGC) large-scale datasets on which we re-trained models, followed by an empirical exploration of the constraint of data removal. Our results show that a dataset presenting between 7 and 30% of outliers benefits from cleaning before training.

Modeling Subject Scoring Behaviors in Subjective Experiments Based on a Discrete Quality Scale

“…It is important to highlight that the AP algorithm utilized by the SUREAL software was endorsed by the ITU in 2021 as the most comprehensive method for subjective quality recovery (as per Section 12.6 of ITU-R P.913 [22]). As the latest standardized approach, the AP algorithm has recently served as the primary benchmark for evaluating newly proposed methods by various authors [12], [31]. Consequently, in the results section, we also consider SUREAL software as the primary benchmarking approach.…”

“…Figure 5 shows the average inconsistency of each subject as defined in Eq (31), as a function of the quality of the stimulus. On average, the proposed subject scoring model estimated lower inconsistency values for stimuli whose quality is in the range from 1 to 1.5 and from 4.5 to 5, compared to what happens in the middle of the quality scale.…”

ZREC: Robust Recovery of Mean and Percentile Opinion Scores

“…Previous work [9] has shown that training objective quality models on cleaned data can improve the prediction performance. In this work, we compared the performance of the 75%SUR prediction model [11] trained on 75%SUR from original datasets without recovery and 75%SUR (25 th percentile) recovered by ZREC and P913-12.6 respectively.…”

“…A range of methods [1,2,3,4,5] with varying complexities have been proposed, and various standards [6,7,8] include recommendations for this purpose. It has also been demonstrated that training learning-based metrics on recovered MOS can enhance their performance to a certain extent [9].…”