Expressiveness Influences Human Vocal Alignment Toward voice-AI

Abstract

This study explores whether people align to expressive speech spoken by a voice-activated artificially intelligent device (voice-AI), specifically Amazon's Alexa. Participants shadowed words produced by the Alexa voice in two acoustically distinct conditions: "regular" and "expressive", containing more exaggerated pitch contours and longer word durations. Another group of participants rated the shadowed items, in an AXB perceptual similarity task, as an assessment of overall degree of vocal alignment. Results show greater vocal alignment toward expressive speech produced by the Alexa voice and, furthermore, systematic variation based on speaker gender. Overall, these findings have applications to the field of affective computing in understanding human responses to synthesized emotional expressiveness.

Comparing alignment toward American, British, and Indian English text-to-speech (TTS) voices: influence of social attitudes and talker guise

“…The Computers Are Social Actors (CASA) theory posits that despite the top-down knowledge that they are communicating with a computer, humans still treat computers as social actors, and behave similarly toward them as they would another person (Nass et al, 1994). A large body of research supports CASA and has shown that humans show similar alignment patterns toward computers as they do in human-human communication (Bell et al, 2003;Branigan et al, 2003;Cohn et al, 2019;Zellou et al, 2021b). These alignment patterns are motivated by linguistic differences and happen at various levels, including syntactically (Branigan et al, 2003;Pearson et al, 2004), phonetically (Cohn et al, 2019;Gessinger et al, 2021;Zellou et al, 2021b), lexically (Branigan et al, 2011;Cowan et al, 2015), and prosodically (Bell et al, 2003;Suzuki and Katagiri, 2007).…”

“…These alignment patterns are motivated by linguistic differences and happen at various levels, including syntactically (Branigan et al, 2003;Pearson et al, 2004), phonetically (Cohn et al, 2019;Gessinger et al, 2021;Zellou et al, 2021b), lexically (Branigan et al, 2011;Cowan et al, 2015), and prosodically (Bell et al, 2003;Suzuki and Katagiri, 2007). Current research has further investigated these phenomena by assessing vocal alignment toward voice-enabled digital assistants (voice artificial intelligence or voice-AI), such as Amazon's Alexa and Apple's Siri (Cohn et al, 2019(Cohn et al, , 2021Zellou and Cohn, 2020;Zellou et al, 2021b;Aoki et al, 2022), and has found evidence that social factors, such as gender (Cohn et al, 2019;Snyder et al, 2019) and conversational role (Zellou et al, 2021b), additionally affect human-computer alignment.…”

“…An open question is whether alignment in human-voice-AI communication directly mirrors alignment patterns in humanhuman communication, or whether different strategies are applied with a device interlocutor. For example, Cohn et al (2019) conducted a shadowing task with human and text-to-speech (TTS) model talkers (using Apple's Siri voices) and investigated gendermediated phonetic alignment for both types of interlocutors. They found that male voices were imitated more than female voices for both human and device model talkers, indicating that similar gender-mediated social patterns of alignment are at play during shadowing with device talkers as with human talkers.…”