ZREC: Robust Recovery of Mean and Percentile Opinion Scores

Zhu, Jingwen; Ak, Ali; Le Callet, Patrick; Sethuraman, Sriram; Rahul, Kumar

doi:10.1109/icip49359.2023.10222033

Cited by 3 publications

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

A Toolkit to Benchmark Point Cloud Quality Metrics with Multi-Track Evaluation Criteria

Ak,

Zerman,

Quach

et al. 2024

2024 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Point clouds (PCs) gained popularity as a representation for 3D objects and scenes and are widely used in numerous applications in augmented and virtual reality domains. Concurrently, quality assessment of PCs became even more relevant to improve various aspects of these imaging pipelines. To stimulate further growth and interest in point cloud quality assessment (PCQA), we created a large-scale PCQA dataset (called "BASICS") which provides the research community with a relevant and challenging dataset to develop reliable objective quality metrics, and we organized the PCVQA grand challenge at ICIP 2023. In this paper, we provide a track-based evaluation methodology for benchmarking visual quality metrics, mirroring the PCVQA grand challenge evaluation scenarios designed to mimic real-life applications. Furthermore, we provide a state-of-theart benchmark for the point cloud quality metrics. The track-based benchmarking approach shows that there is room for improvement in certain research directions, drawing attention to open problems in the PCQA domain.

show abstract

A Toolkit to Benchmark Point Cloud Quality Metrics with Multi-Track Evaluation Criteria

Ak,

Zerman,

Quach

et al. 2024

2024 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

Comparison of Crowdsourcing And Laboratory Settings for Subjective Assessment of Video Quality and Acceptability & Annoyance

Ak,

Gera,

Noyes

et al. 2024

2024 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

User satisfaction is significantly influenced by their expectations of video quality. Even when users are presented with identical video stimuli, the Quality of Experience (QoE) can vary based on the context. The acceptability and annoyance paradigm serves as a tool to understand this relationship by measuring QoE as a function of user expectations and video quality. Traditionally, subjective experiments assessing QoE have been conducted in controlled laboratory settings. While the extension of traditional video quality experiments to crowdsourcing settings is well-explored, the impact of crowdsourcing on QoE studies has not been thoroughly examined. This study explore the potential use of crowdsourcing platforms for acceptability & annoyance experiments. To this end, video quality and acceptability & annoyance experiments were conducted in both laboratory and crowdsourcing settings. The findings reveal a more linear relationship between video quality and QoE in crowdsourcing settings. Subjects in crowdsourcing settings tend to have higher expectations of video quality, resulting in a slight increase in acceptability & annoyance thresholds compared to laboratory experiments. Analyses suggest that extending acceptability & annoyance experiments to crowdsourcing is not as straightforward as extending traditional video quality experiments. In crowdsourcing settings, priming subject expectations with instructions is not as effective as it is in laboratory conditions.

show abstract

BASICS: Broad Quality Assessment of Static Point Clouds in a Compression Scenario

Ak,

Zerman,

Quach

et al. 2024

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Point clouds have become increasingly prevalent in representing 3D scenes within virtual environments, alongside 3D meshes. Their ease of capture has facilitated a wide array of applications on mobile devices, from smartphones to autonomous vehicles. Notably, point cloud compression has reached an advanced stage and has been standardized. However, the availability of quality assessment datasets, which are essential for developing improved objective quality metrics, remains limited. In this paper, we introduce BASICS, a large-scale quality assessment dataset tailored for static point clouds. The BASICS dataset comprises 75 unique point clouds, each compressed with four different algorithms including a learning-based method, resulting in the evaluation of nearly 1500 point clouds by 3500 unique participants. Furthermore, we conduct a comprehensive analysis of the gathered data, benchmark existing point cloud quality assessment metrics and identify their limitations. By publicly releasing the BASICS dataset, we lay the foundation for addressing these limitations and fostering the development of more precise quality metrics.

show abstract

ZREC: Robust Recovery of Mean and Percentile Opinion Scores

Cited by 3 publications

References 15 publications

A Toolkit to Benchmark Point Cloud Quality Metrics with Multi-Track Evaluation Criteria

A Toolkit to Benchmark Point Cloud Quality Metrics with Multi-Track Evaluation Criteria

Comparison of Crowdsourcing And Laboratory Settings for Subjective Assessment of Video Quality and Acceptability & Annoyance

BASICS: Broad Quality Assessment of Static Point Clouds in a Compression Scenario

Contact Info

Product

Resources

About