Models of dataset size, question design, and cross-language speech perception for speech crowdsourcing applications

Abstract

Transcribers make mistakes. Workers recruited in a crowdsourcing marketplace, because of their varying levels of commitment and education, make more mistakes than workers in a controlled laboratory setting. Methods for compensating transcriber mistakes are desirable because, with such methods available, crowdsourcing has the potential to significantly increase the scale of experiments in laboratory phonology. This paper provides a brief tutorial on statistical learning theory, introducing the relationship between dataset size and estimation error, then presents a theoretical description and preliminary results for two new methods that control labeler error in laboratory phonology experiments. First, we discuss the method of crowdsourcing over error-correcting codes. In the error-correcting-code method, each difficult labeling task is first factored, by the experimenter, into the product of several easy labeling tasks (typically binary). Factoring increases the total number of tasks, nevertheless it results in faster completion and higher accuracy, because workers unable to perform the difficult task may be able to meaningfully contribute to the solution of each easy task. Second, we discuss the use of explicit mathematical models of the errors made by a worker in the crowd. In particular, we introduce the method of mismatched crowdsourcing, in which workers transcribe a language they do not understand, and an explicit mathematical model of second-language phoneme perception is used to learn and then compensate their transcription errors. Though introduced as technologies that increase the scale of phonology experiments, both methods have implications beyond increased scale. The method of easy questions permits us to probe the perception, by untrained listeners, of complicated phonological models; examples are provided from the prosody of English and Hindi. The method of mismatched crowdsourcing permits us to probe, in more detail than ever before, the perception of phonetic categories by listeners with a different phonological system.

Controlling Prominence Realisation in Parametric DNN-Based Speech Synthesis

“…The development of the speech tagging tools in turn relies on the knowledge on how human listeners -the users of TTS systems -perceive and categorise prominence [1,7]. Recent advances show that crowdsourcing methods enable to directly access human prominence judgments in a relatively short time [8,9,10].…”

Towards a replication culture in phonetic research: Speech production research in the classroom

“…Despite methodological debates surrounding the trade-off between experimental control and stylistic variation (Xu 2010;Wagner et al 2015 and references therein), an increasing number of speech scientist use crowdsourcing platforms to collect linguistic data (Hasegawa-Johnson, Cole, Jyothi, & Varshney 2015). This suggests that researchers are willing to trade an increased level of variability and lack of control with quick and convenient access to large amounts of data.…”

Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?

“…As an illustrative example, consider the problem of mismatched crowdsourcing for speech transcription, which has garnered interest in the signal processing community [4,6,9,12,14,23]. Suppose the four possibilities for a velar stop consonant to transcribe are R = { , , , }.…”