HodgeRank on Random Graphs for Subjective Video Quality Assessment

Xu, Qianqian; Huang, Qingming; Jiang, Tingting; Yan, Bowei; Wang, Lin; Yao, Yuan

doi:10.1109/tmm.2012.2190924

Cited by 67 publications

(58 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robust Learning to rank Statistical ranking has been widely studied in statistics [20,10] and computer science [44,45]. By aggregating pairwise local rankings into a global ranking, methods such as Huber-LASSO [46,18] have the potential to be robust against local ranking noise [5,31].…”

Section: Related Workmentioning

confidence: 99%

Interestingness Prediction by Robust Learning to Rank

Hospedales

Xiang

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. The problem of predicting image or video interestingness from their low-level feature representations has received increasing interest. As a highly subjective visual attribute, annotating the interestingness value of training data for learning a prediction model is challenging. To make the annotation less subjective and more reliable, recent studies employ crowdsourcing tools to collect pairwise comparisons -relying on majority voting to prune the annotation outliers/errors. In this paper, we propose a more principled way to identify annotation outliers by formulating the interestingness prediction task as a unified robust learning to rank problem, tackling both the outlier detection and interestingness prediction tasks jointly. Extensive experiments on both image and video interestingness benchmark datasets demonstrate that our new approach significantly outperforms state-of-the-art alternatives.

show abstract

Section: Related Workmentioning

confidence: 99%

Interestingness Prediction by Robust Learning to Rank

Hospedales

Xiang

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…While the commonly used five point MOS scale requires a training of the test participants to correctly evaluate the upper and lower baseline of the available samples, a pair-wise comparison is usually easier to understand. However, a pair-wise comparison of huge data sets is usually not possible, but approaches like the HodgeRank on Random Graphs [25] can be used to derive results from incomplete and imbalances comparison sets, extended in [26] for crowdsourcing-based assessment during which comparisons are produced consecutively. This version of the HodgeRank on Randoms Graphs is able to update the results online instead of working batch wise on the complete sample test.…”

Section: G Other Tools and Approachesmentioning

confidence: 99%

Survey of web-based crowdsourcing frameworks for subjective quality assessment

Hoßfeld

Hirth

Korshunov

et al. 2014

2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

Abstract-The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.

show abstract

“…Paired comparison takes advantage of simple comparative judgements to prioritize a set of stimuli, where subjects' preferences for the stimuli can be quantified via probabilistic choice modeling [17,72]. This technique is used in various domains, notably decision making and psychometric testing.…”

Section: B Paired Comparisonmentioning

confidence: 99%

“…They demonstrated that their scheme can provide reliable QoE estimates based on users' judgements on merely 29% of all possible pairs. In addition, Xu et al [72] further explored how to efficiently select pairs that require users' preference judgements the most. They proposed a random partial paired comparison approach based on random graph theory and Hodge theory, and showed that the complexity O(n 2 ) of "traditional" paired comparisons can be reduced to O(n 1.5 ) without significantly sacrificing the accuracy of estimated quality.…”

Section: B Issues With Paired Comparisonmentioning

confidence: 99%

Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Chen

Chang

et al. 2013

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Crowdsourcing has emerged in recent years as a potential strategy to enlist the general public to solve a wide variety of tasks. With the advent of ubiquitous Internet access, it is now feasible to ask an Internet crowd to conduct QoE (Quality of Experience) experiments on their personal computers in their own residences rather than in a laboratory. The considerable size of the Internet crowd allows researchers to crowdsource their experiments to a more diverse set of participant pool at a relatively low economic cost. However, as participants carry out experiments without supervision, the uncertainty of the quality of their experiment results is a challenging problem.In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content. To overcome the aforementioned quality problem, we employ a paired comparison method in our framework. The advantages of our framework are: 1) trustworthiness due to the support for cheat detection; 2) a simpler rating procedure than that of the commonly-used but more difficult mean opinion score (MOS), which places less burden on participants; 3) economic feasibility since reliable QoE measures can be acquired with less effort compared with MOS; and 4) generalizability across a variety of multimedia content. We demonstrate the effectiveness and efficiency of the proposed framework by a comparison with MOS. Moreover, the results of four case studies support our assertion that the framework can provide reliable QoE evaluation at a lower cost.

show abstract

HodgeRank on Random Graphs for Subjective Video Quality Assessment

Cited by 67 publications

References 31 publications

Interestingness Prediction by Robust Learning to Rank

Interestingness Prediction by Robust Learning to Rank

Survey of web-based crowdsourcing frameworks for subjective quality assessment

Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Contact Info

Product

Resources

About