Perceptual Ratings of Voice Likability Collected Through In-Lab Listening Tests vs. Mobile-Based Crowdsourcing

Abstract

Human perceptions of speaker characteristics, needed to perform automatic predictions from speech features, have generally been collected by conducting demanding in-lab listening tests under controlled conditions. Concurrently, crowdsourcing has emerged as a valuable approach for running user studies through surveys or quantitative ratings. Micro-task crowdsourcing markets enable the completion of small tasks (commonly of minutes or seconds), rewarding users with micro-payments. This paradigm permits effortless collection of user input from a large and diverse pool of participants at low cost. This paper presents different auditory tests for collecting perceptual voice likability ratings employing a common set of 30 male and female voices. These tests are based on direct scaling and on paired-comparisons, and were conducted in the laboratory and via crowdsourcing using micro-tasks. Design considerations are proposed for adapting the laboratory listening tests to a mobile-based crowdsourcing platform to obtain trustworthy listeners' answers. Our likability scores obtained by the different test approaches are highly correlated. This outcome motivates the use of crowdsourcing for future listening tests investigating e.g. speaker characterization, reducing the efforts involved in engaging participants and administering the tests on-site.

Spot the Pleasant People! Navigating the Cocktail Party Buzz

“…Voice likability has been described as "the sound of her/his voice and manner of speaking" [4], or "a speaker social characteristic that can determine the listener's attitudes and decisions towards speakers and their message." [5]. Both definitions make the point that voice likability does not only concern the speakers' inherent voice characteristics, but the manner in which speakers intentionally or unintentionally communicate their message.…”

“…Voice likability studies typically use a Likert scale from 1 to 5, 7 or 9 (e.g. [5]- [10]), but there are also examples of binary (non-likable or likable) or trinary likability (non-likable, neutral or likable) likability assessment [11]. In these studies, respondents judge voices sequentially, one at a time, which makes evaluations of many voice samples tedious and timeconsuming.…”

“…In other studies, different voices are pitted against each other and their likability compared, sometimes after separating female and male voices (e.g. [5]). [5] also shows that comparisons between likability scores obtained in laboratory settings correlate well with scores from crowdsourcing.…”

“…[5]). [5] also shows that comparisons between likability scores obtained in laboratory settings correlate well with scores from crowdsourcing. There is also an increasing amount of work in which spectral parameters are used to predict voice likability automatically (e.g.…”