Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Pérez, Pablo; Janowski, Lucjan; García, Narciso; Pinson, Margaret H.

doi:10.1109/tmm.2021.3098450

Cited by 9 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from keeping the number of participants low due to opportunitylimited situations, the recent developments on novel immersive media addressing a wide range of XR modes justify revisiting also the role of pilot studies with a few participants in an overall experimental design. In particular, in [20], the few observers with repetitions (FOWR) subjective test protocol has been proposed that engages four to six team member each rating the test stimuli several times. To reveal the suitability of the FOWR protocol for subjective quality assessment, a subjective test was conducted in which the participants were instructed to 10 times repeat the experiment of rating 110 processed video sequences.…”

Section: Subjective Tests Under Opportunity-limited Conditionsmentioning

confidence: 99%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

View full text Add to dashboard Cite

Immersive media such as virtual reality, augmented reality, and 360◦ video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360◦ video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent with each user having their own behavior signature. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360◦ video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360◦ video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community.

show abstract

Section: Subjective Tests Under Opportunity-limited Conditionsmentioning

confidence: 99%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

View full text Add to dashboard Cite

show abstract

“…In addition to reducing bias and noise, meta-datasets simplify the NR metric training process. However, [93] challenges the concept of anchor conditions or a "common set" in cross-lab experiments. Inclusion of a common set impacted both the evaluation of the common set and the evaluation of the media in the new experiment.…”

Section: B Impact Of Dataset Design On Nr Metricsmentioning

confidence: 99%

Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility

Pinson

2023

IEEE Trans. on Broadcast.

Self Cite

View full text Add to dashboard Cite

This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.

show abstract

“…To test GSD's generalizability capabilities in practice, we use data from four MQA studies: (i) MM2 [22], (ii) VQEG HDTV Phase I [21], (iii) NFLX (cf. Section II-C to learn more about this study) and (iv) ITERO [25]. From these studies, we extract responses for selected stimuli.…”

Section: Better Than Empirical Distributionmentioning

confidence: 99%

Generalized Score Distribution: A Two-Parameter Discrete Distribution Accurately Describing Responses From Quality of Experience Subjective Experiments

Nawała

Janowski

Ćmiel

et al. 2023

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Subjective responses from Multimedia Quality Assessment (MQA) experiments are conventionally analyzed with methods not suitable for the data type these responses represent. Furthermore, obtaining subjective responses is resource intensive. Thus, a method that allows the reuse of existing responses would be beneficial. Applying improper data analysis methods leads to difficulty in interpreting results. This increases the probability of drawing erroneous conclusions. Building upon existing subjective responses is resource friendly and helps develop machine learning (ML) based visual quality predictors. In this work, we show that using a discrete model for analyzing responses from MQA subjective experiments is feasible. We indicate that our proposed Generalized Score Distribution (GSD) properly describes response distributions observed in typical MQA experiments. We also highlight interpretability of GSD parameters and indicate that the GSD outperforms the approach based on sample empirical distribution when it comes to bootstrapping. Furthermore, we provide evidence that the GSD outcompetes the state-of-theart model both in terms of goodness-of-fit and bootstrapping capabilities. To accomplish the aforementioned objectives, we analyze more than one million subjective responses from over 30 subjective experiments.

show abstract

Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

Cited by 9 publications

References 25 publications

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility

Generalized Score Distribution: A Two-Parameter Discrete Distribution Accurately Describing Responses From Quality of Experience Subjective Experiments

Contact Info

Product

Resources

About