On the Opinion Score Consistency in Repeated 360° Video Quality Assessment for Standing and Seated Viewing on Head-Mounted Displays

Elwardy, Majed; Zepetnick, Hans-Jurgen; Chu, Thi My Chinh; Hu, Yan

doi:10.1109/icspcs53099.2021.9660331

Cited by 3 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In continuation of our preliminary work and exploratory findings reported in [21,22], this article provides a comprehensive study on participants' consistency in repeated subjective tests on 360 • video quality assessment regarding opinion scores and head movements. In particular, in this article, the results of a comprehensive statistical analysis of different sets of opinion scores, average opinion scores, accumulated opinion scores, and head movements are presented using visualizations that offer more information such as violin plots and cumulative distribution functions (CDFs).…”

Section: Contributionsmentioning

confidence: 85%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

Self Cite

View full text Add to dashboard Cite

Immersive media such as virtual reality, augmented reality, and 360◦ video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360◦ video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent with each user having their own behavior signature. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360◦ video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360◦ video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community.

show abstract

Section: Contributionsmentioning

confidence: 85%

On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Elwardy,

Zepernick,

Chu

et al. 2024

EAI Endorsed Trans Ind Net Intel Syst

Self Cite

View full text Add to dashboard Cite

show abstract

“…This section describes the experimental setup used in our subjective test campaigns on 360 • video quality assessment on an HMD [17]- [23] to the extent needed for the understanding of the ACR360 dataset. More information about test procedures, 360 • video processing, generation of the test stimuli, and specifcations of the software suites and hardware equipment used in the various subjective tests can be found in [17].…”

Section: Methodsmentioning

confidence: 99%

ACR360: A Dataset on Subjective 360° Video Quality Assessment Using ACR Methods

Elwardy,

Zepernick,

et al. 2023

2023 16th International Conference on Signal Processing and Communication System (ICSPCS)

Self Cite

View full text Add to dashboard Cite

The recent advances in immersive technologies have been essential in the development of a wide range of novel standalone and networked immersive media applications. The concepts of virtual reality, augmented reality, and mixed reality relate to different compositions of real and computer-generated virtual objects. In this context, 360 • video streaming has become increasingly popular offering improved immersive experiences when viewed on a head-mounted display (HMD). An important component in the development of novel immersive media systems are subjective tests in which participants assess the quality of experience of representative test stimuli. In this paper, the annotated ACR360 dataset is presented which is publicly available on GitHub. The ACR360 dataset contains a wide range of psychophysical and psychophysiological data that was collected in subjective tests on 360 • video quality. The test stimuli were shown on an HMD and rated according to the absolute category rating (ACR) and modifed ACR (MACR) methods. To support an easy exploration and utilization of the ACR360 dataset by the research community, its structure on GitHub is described and a comprehensive illustration of analysis options are provided for each data category. The ACR360 dataset may be used for conducting meta-analysis in combination with other datasets to improve precision and to pursue research questions that cannot be answered by an individual study.

show abstract