Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study

Abstract

Human speech production requires the dynamic regulation of air through the vocal system. While virtual character systems commonly are capable of speech output, they rarely take breathing during speaking-speech breathing-into account. We believe that integrating dynamic speech breathing systems in virtual characters can significantly contribute to augmenting their realism. Here, we present a novel control architecture aimed at generating speech breathing in virtual characters. This architecture is informed by behavioral, linguistic and anatomical knowledge of human speech breathing. Based on textual input and controlled by a set of lowand high-level parameters, the system produces dynamic signals in real-time that control the virtual character's anatomy (thorax, abdomen, head, nostrils, and mouth) and sound production (speech and breathing). In addition, we perform a study to determine the effects of including breathing-motivated speech movements, such as head tilts and chest expansions during dialogue on a virtual character, as well as breathing sounds. This study includes speech that is generated both from a text-to-speech engine as well as from recorded voice.

Evaluating the Influence of Phoneme-Dependent Dynamic Speaker Directivity of Embodied Conversational Agents' Speech

“…Presence has been bolstered, for example, by adding soundscapes or step sound [12], or a more advanced speech auralization [8]. Some research has been conducted to increase the naturalism by introducing artificial breathing during speech [6,22], as well as phoneme-dependent lip syncing and face animations [23,28]. Mehra and colleagues [15] suggest that sound source directivities may improve the realism of auralizations.…”

Breathing and Speech Planning in Spontaneous Speech Synthesis

“…Breathing not only adds to the perceived naturalness of synthetic speech over using silent pauses [3], but also breaks the message into smaller chunks that may be easier to comprehend [4] and improves recall [5]. Moreover, TTS systems that integrate breathing into the synthesis and allow it to be controlled have the potential to make virtual characters [6] and social robots more convincing.…”