Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Abstract

Crowdsourcing has emerged in recent years as a potential strategy to enlist the general public to solve a wide variety of tasks. With the advent of ubiquitous Internet access, it is now feasible to ask an Internet crowd to conduct QoE (Quality of Experience) experiments on their personal computers in their own residences rather than in a laboratory. The considerable size of the Internet crowd allows researchers to crowdsource their experiments to a more diverse set of participant pool at a relatively low economic cost. However, as participants carry out experiments without supervision, the uncertainty of the quality of their experiment results is a challenging problem.In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content. To overcome the aforementioned quality problem, we employ a paired comparison method in our framework. The advantages of our framework are: 1) trustworthiness due to the support for cheat detection; 2) a simpler rating procedure than that of the commonly-used but more difficult mean opinion score (MOS), which places less burden on participants; 3) economic feasibility since reliable QoE measures can be acquired with less effort compared with MOS; and 4) generalizability across a variety of multimedia content. We demonstrate the effectiveness and efficiency of the proposed framework by a comparison with MOS. Moreover, the results of four case studies support our assertion that the framework can provide reliable QoE evaluation at a lower cost.

Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing

“…Studies from literature have shown that using crowdtesting for the QoE assessment of a wide range of video applications can deliver results similar to traditional testing in the lab environment: Keimel et al have shown in [21], [23] that crowdtesting delivers results within the acceptable inter-lab variation between different testing labs for standard conforming QoE assessment, Chen et al discussed crowdtesting for audio-visual QoE of Internet-based applications in [24], [25], which was discussed more in detail by Wu et al in [26], and Hossfeld et al applied crowdtesting to the influence of stalling events [20] and initial delays [27] on the QoE in video streaming applications. For pairwise comparison QoE tests, Xu et al.…”

Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario

“…Two frameworks often utilised are the Quadrant of Euphoria by Chen et al [14,20,21] and the QualityCrowd framework by Keimel et al [16] that is also used in this contribution. For a detailed discussion about web-based crowdsourcing frameworks for subjective quality assessment we refer to the survey in [22].…”

Survey of web-based crowdsourcing frameworks for subjective quality assessment

“…The Quadrant of Euphoria is a web-based online crowdsourcing platform for the QoE evaluation of audio, visual, and audio-visual stimuli proposed by Chen et al in [12], [13] and extended by Wu et al in [14] .…”