Influence of Number of Stimuli for Subjective Speech Quality Assessment in Crowdsourcing

Jiménez, Rafael Zequeira; Gallardo, Laura Fernández; Möller, Sebastian

doi:10.1109/qomex.2018.8463298

Cited by 18 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, several bodies of research successfully employed CS to collect speech quality scores of a standard speech database (e.g. [10][11][12]). Due to the adoption of CS as a test paradigm, the analysis of the differences between the lab-based and CS-based tests has been of interest in multiple studies [13].…”

Section: Related Workmentioning

confidence: 99%

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

et al. 2020

View full text Add to dashboard Cite

Subjective speech quality assessment has traditionally been carried out in laboratory environments under controlled conditions. With the advent of crowdsourcing platforms tasks, which need human intelligence, can be resolved by crowd workers over the Internet. Crowdsourcing also offers a new paradigm for speech quality assessment, promising higher ecological validity of the quality judgments at the expense of potentially lower reliability. This paper compares laboratory-based and crowdsourcing-based speech quality assessments in terms of comparability of results and efficiency. For this purpose, three pairs of listening-only tests have been carried out using three different crowdsourcing platforms and following the ITU-T Recommendation P.808. In each test, listeners judge the overall quality of the speech sample following the Absolute Category Rating procedure. We compare the results of the crowdsourcing approach with the results of standard laboratory tests performed according to the ITU-T Recommendation P.800. Results show that in most cases, both paradigms lead to comparable results. Notable differences are discussed with respect to their sources, and conclusions are drawn that establish practical guidelines for crowdsourcing-based speech quality assessment.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Regarding the experimental methodologies used in crowdsourcing, most CS studies have used the pairwise comparison (PC) method as it is straightforward: the task of choosing one of the two stimuli is simpler than rating them on a discrete or continuous scale [9], [11], [12]. Other works have adopted the Absolute Category Rating (ACR) method [3], [10].…”

Section: Introductionmentioning

confidence: 99%

Exploring Crowdsourcing for Subjective Quality Assessment of 3D Graphics

Nehmé

Callet

Dupont

et al. 2021

2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP)

View full text Add to dashboard Cite

Multimedia subjective quality assessment experiments are the most prominent and reliable way to evaluate the visual quality as perceived by human observers. Along with laboratory (lab) subjective experiments, crowdsourcing (CS) experiments have become very popular in recent years, e.g., during the COVID-19 pandemic these experiments provide an alternative to lab tests. However, conducting subjective quality assessment tests in CS raises many challenges: internet connection quality, lack of control on participants' environment, participants' consistency and reliability, etc. In this work, we evaluate the performance of CS studies for 3D graphics quality assessment. To this end, we conducted a CS experiment based on the double stimulus impairment scale method and using a dataset of 80 meshes with diffuse color information corrupted by various distortions. We compared its results with those previously obtained in a lab study conducted on the same dataset and in a virtual reality environment. Results show that under controlled conditions and with appropriate participant screening strategies, a CS experiment can be as accurate as a lab experiment.

show abstract

“…listening tests that are conducted over a web service such as Amazon Mechanical Turk. In these cases the quality of ratings can be worse than traditional lab tests and may require additional prescreening and restrictions [19], (e.g. raters that always give the highest or lowest score due to any number of reasons including bad headphones or malicious ratings), and outliers will have an undesirably large influence on the mapping.…”

Section: Introductionmentioning

confidence: 99%

Marginal Effects of Language and Individual Raters on Speech Quality Models

Chinen

2021

IEEE Access

View full text Add to dashboard Cite

Speech quality is often measured via subjective testing, or with objective estimators of mean opinion score (MOS) such as ViSQOL or POLQA. Typical MOS-estimation frameworks use signal level features but do not use language features that have been shown to have an effect on opinion scores. If there is a conditional dependence between score and language given these signal features, introducing language and rater predictors should provide a marginal improvement in predictions. The proposed method uses Bayesian models that predict the individual opinion score instead of MOS. Several models that test various combinations of predictors were used, including predictors that capture signal features, such as frequency band similarity, as well as features that are related to the listener, such as a language and rater index. The models are fit to the ITU-T P. Supplement 23 dataset, and posterior samples are drawn from distributions of both the model parameters and the resulting opinion score outcomes. These models are compared to MOS models by integrating over posterior samples per utterance. An experiment was conducted by ablating different predictors for several types of Bayesian hierarchical models (including ordered logistic and truncated normal individual score distributions, as well as MOS distributions) to find the marginal improvement of language and rater. The models that included language and/or rater obtained significantly lower errors (0.601 versus 0.684 root-mean-square error (RMSE)) and higher correlation. Additionally, individual rater models matched or exceeded the performance of MOS models.

show abstract

Influence of Number of Stimuli for Subjective Speech Quality Assessment in Crowdsourcing

Cited by 18 publications

References 9 publications

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods

Exploring Crowdsourcing for Subjective Quality Assessment of 3D Graphics

Marginal Effects of Language and Individual Raters on Speech Quality Models

Contact Info

Product

Resources

About