A Large-Scale User Study of an Alexa Prize Chatbot: Effect of TTS Dynamism on Perceived Quality of Social Dialog

Abstract

This study tests the effect of cognitiveemotional expression in an Alexa text-tospeech (TTS) voice on users' experience with a social dialog system. We systematically introduced emotionally expressive interjections (e.g., "Wow!") and filler words (e.g., "um", "mhmm") in an Amazon Alexa Prize socialbot, Gunrock. We tested whether these TTS manipulations improved users' ratings of their conversation across thousands of real user interactions (n=5,527). Results showed that interjections and fillers each improved users' holistic ratings, an improvement that further increased if the system used both manipulations. A separate perception experiment corroborated the findings from the user study, with improved social ratings for conversations including interjections; however, no positive effect was observed for fillers, suggesting that the role of the rater in the conversation-as active participant or external listener-is an important factor in assessing social dialogs.

Prosodic Differences in Human- and Alexa-Directed Speech, but Similar Local Intelligibility Adjustments

“…For instance, here people are reacting to emotional expressiveness by both types of interlocutors similarly. This explanation is consistent with work showing similar affective responses to computers as seen in humanhuman interaction (e.g., Brave et al, 2005;Cohn et al, 2019a;.…”

“…For instance, Brave and colleagues (2005) found when computer systems expressed empathetic emotion, they were rated more positively. For voice-AI, there is a growing body of work testing how individuals perceive emotion in TTS voices (Cohn et al, 2019a;Cohn et al, 2020a). For example, an Amazon Alexa Prize socialbot was rated more positively if it used emotional interjections (Cohn et al, 2019a).…”

“…For voice-AI, there is a growing body of work testing how individuals perceive emotion in TTS voices (Cohn et al, 2019a;Cohn et al, 2020a). For example, an Amazon Alexa Prize socialbot was rated more positively if it used emotional interjections (Cohn et al, 2019a). Alternatively, the presence of emotionality might lead to distinct clear speech strategies for the human and voice-AI interlocutors.…”

Gunrock: A Social Bot for Complex and Engaging Long Conversations

“…We use a rule-based system to systematically add interjections, specifically Alexa Speechcons, and fillers to approximate human-like cognitive-emotional expression (Tokuhisa and Terashima, 2006). For more on the framework and analysis of the TTS modifications, see (Cohn et al, 2019).…”

An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog

“…Conversation analysis is central to NLP and multiple approaches have analyzed this dialog structure (Jurafsky et al, 1998;Pareti and Lando, 2018;Cohn et al, 2019) and developed conversational agents to engage with people (e.g., Fang et al, 2018;Xu et al, 2020;Hong et al, 2020). Recent work has focused on generating open domain social chatbots that engage in sustained conversations in a natural way (Ram et al, 2018).…”