Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Abstract

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment.

An Open Source Implementation of ITU-T Recommendation P.808 with Validation

“…We kept the structure of the P.808 Toolkit the same; further details about the P.808 Toolkit can be found in [3] and details on validation of the ITU-T Rec. P.808 in [5].…”

Crowdsourcing Approach for Subjective Evaluation of Echo Impairment

“…Further details about the P.808 Toolkit can be found in [11] and details on validation of the ITU-T Rec. P.808 in [16].…”

“…Further details about the P.808 Toolkit can be found in [15] and details on validation of the ITU-T Rec. P.808 in [20]. The test participant (marked by "You") is positioned in the center of the communication.…”

“…PESQ has also been shown to not have a high correlation to subjective speech quality in the presence of background noise [8]. Traditionally, the Quality of Experience (QoE) is measures by con-ducting subjective test in the laboratory following the standardized test methods [9]. The ITU-T Rec.…”

More for Less: Non-Intrusive Speech Quality Assessment with Limited Annotations

“…Annotating large datasets is a general problem in machine learning, especially in multimedia quality assessment where multiple recruiters are needed to annotate only one stimulus. Although annotations obtained through crowdsourcing can be as valid as a lab setting quality measures [4], [5], annotating data is still a costly and time-consuming operation.…”