On the Impact of COVID-19 on Subjective Digital Media Quality Assessment

Zepernick, Hans‐Jürgen; Pieper, Kerstin; Spang, Robert P.; Engelke, Ulrich; Hirth, Matthias; Naderi, Babak

doi:10.1109/mmsp53017.2021.9733583

Cited by 5 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An aim of this experimental design was to keep the time for the participants being required in the laboratory to a minimum. In [19], the impact of COVID- Repeated Subjective Tests with Few Participants. Given the need for subjective tests on immersive media quality assessment that engage only a few participants, related research on the number of participants has regained interest in recent years.…”

Section: Subjective Tests Under Opportunity-limited Conditionsmentioning

confidence: 99%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

View full text Add to dashboard Cite

Immersive media such as virtual reality, augmented reality, and 360◦ video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360◦ video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent with each user having their own behavior signature. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360◦ video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360◦ video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community.

show abstract

Section: Subjective Tests Under Opportunity-limited Conditionsmentioning

confidence: 99%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

View full text Add to dashboard Cite

show abstract

“…IKN − 1 n=1 k=1 i=1 design proposed in [14] as an option to conduct subjective C. Standard Deviation of Opinion Score Hypothesis tests under pandemic conditions.…”

Section: Of Test Stimuli Follows the Alternative Experimentalmentioning

confidence: 99%

“…In view of the current pandemic situation, to keep hygiene related concerns to a minimum, two participants, referred to as P1 and P2, were engaged in the repeated subjective tests as in [13]. Following the suggestion of [14] to consider time averages instead of ensemble averages, each participant assessed a sequence of a large number of test stimuli in standing and seated viewing, i.e., a total of 720 different 360 • videos covering a wide range of qualities, rather than many participants assessing a small number of test stimuli. This approach is also in line with the suggestion in [15] to consider in-person experiments with a low sample size during pandemic situations that reveal new research avenues and indicate the need for follow-up experiments with a larger sample size.…”

Section: Introductionmentioning

confidence: 99%

On the Opinion Score Consistency in Repeated 360° Video Quality Assessment for Standing and Seated Viewing on Head-Mounted Displays

Elwardy

Zepetnick

Chu

et al. 2021

2021 15th International Conference on Signal Processing and Communication Systems (ICSPCS)

View full text Add to dashboard Cite

The development of immersive media systems and services relies on subjective tests that provide ground truth on the quality of experience as perceived by humans. In this paper, we investigate the opinion score consistency through a repeated subjective 360 • video quality assessment experiment. The test stimuli are presented on a head-mounted display with participants given the task to rate the video quality in three repeated sessions for both standing and seated viewing. The statistical analysis of the data from the subjective tests aims at revealing if participants change their rating behavior over time or keep their opinion scores given to the test stimuli in standing and seated viewing consistent in a statistical sense. The experimental results are reported in terms of histograms of opinion scores, skewness and kurtosis of opinion scores, mean opinion scores, standard deviation of opinion scores (SOS), SOS ftting functions, and analysis of variance tests. The statistical analysis supports the conjecture that each participant has its own but consistent rating behavior throughout the respective sessions for standing and seated viewing. The fnding that the quality rating behavior of the individual participant does not fundamentally change over time may, e.g., assist in scheduling subjective tests under pandemic conditions where experimental campaigns may need to be stalled for an unknown period of time.

show abstract

“…As such, publicly available annotated datasets are needed as a foundation to perform perceptionbased quality assessment of algorithms, systems, and services as well as to develop objective perceptual quality models. Furthermore, in view of pandemic situations and related implications on conducting subjective tests for digital media quality assessment, it has been suggested in [17] to share work and data from subjective tests, pilot studies, and self-report studies in the spirit of the open science movement. This includes publishing trial protocols, stimuli material, analysis scripts, raw data, and performing insightful meta-analyses on available datasets as an effcient means to alleviate the diffculties of conducting subjective tests under pandemic conditions.…”

Section: Introductionmentioning

confidence: 99%

SSV360: A Dataset on Subjetive Quality Assessment of 360° Videos for Standing and Seated Viewing on an HMD

Elwardy

Zepernick

2022

2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce the SSV360 dataset containing psychophysical and psychophysiological data gathered in a pilot study on the effect of standing and seated viewing of 360 • videos on subjective quality assessment. The stimuli were shown to the participants on a head-mounted display for both viewing conditions. The ground truth contained in the SSV360 dataset may support, e.g., benchmarking of immersive media processing algorithms and developing of objective perceptual quality metrics.

show abstract

On the Impact of COVID-19 on Subjective Digital Media Quality Assessment

Cited by 5 publications

References 22 publications

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

On the Opinion Score Consistency in Repeated 360° Video Quality Assessment for Standing and Seated Viewing on Head-Mounted Displays

SSV360: A Dataset on Subjetive Quality Assessment of 360° Videos for Standing and Seated Viewing on an HMD

Contact Info

Product

Resources

About