Augmented CycleGANs for Continuous Scale Normal-to-Lombard Speaking Style Conversion

Abstract

Lombard speech is a speaking style associated with increased vocal effort that is naturally used by humans to improve intelligibility in the presence of noise. It is hence desirable to have a system capable of converting speech from normal to Lombard style. Moreover, it would be useful if one could adjust the degree of Lombardness in the converted speech so that the system is more adaptable to different noise environments. In this study, we propose the use of recently developed Augmented cycleconsistent adversarial networks (Augmented CycleGANs) for conversion between normal and Lombard speaking styles. The proposed system gives a smooth control on the degree of Lombardness of the mapped utterances by traversing through different points in the latent space of the trained model. We utilize a parametric approach that uses the Pulse Model in Log domain (PML) vocoder to extract features from normal speech that are then mapped to Lombard-style features using the Augmented CycleGAN. Finally, the mapped features are converted to Lombard speech with PML. The model is trained on multi-language data recorded in different noise conditions, and we compare its effectiveness to a previously proposed CycleGAN system in experiments for intelligibility and quality of mapped speech.

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

“…Wang et al, 2018) and have also started to be applied to speech transformations. For example, GANs were recently used to transform a voice into its Lombard counterpart (a particular type of vocal effort which makes the voice more intelligible in background noise; Seshadri, Juvela, Alku, & Räsänen, 2019). All such advances open exciting new possibilities to create emotional voice and speech transformations, which will certainly find their way to the community in the upcoming years.…”

“…Finally, generative adversarial networks (GANs), a special class of DNN architecture capable of learning a deterministic mapping from one style of stimulus to another (Goodfellow et al, 2014), are increasingly used to create visual transformations (e.g., smiles; W. Wang et al, 2018) and have also started to be applied to speech transformations. For example, GANs were recently used to transform a voice into its Lombard counterpart (a particular type of vocal effort which makes the voice more intelligible in background noise; Seshadri, Juvela, Alku, & Räsänen, 2019). All such advances open exciting new possibilities to create emotional voice and speech transformations, which will certainly find their way to the community in the upcoming years.…”

Enhancing Speech Intelligibility in Text-To-Speech Synthesis Using Speaking Style Conversion

“…To overcome this, deep neural network approaches were implemented where the robustness of acoustic modeling is improved by efficient mapping between linguistic and acoustic features. Inspired by the success of adversarial generative models, Cycle-consistent adversarial networks (CycleGANs) showed promising results in terms of speech quality and the magnitude of the perceptual change between speech styles [11,12]. An extension to recurrent neural networks and particularly long short-term memory networks (LSTMs) were proposed that it successfully adapted normal speaking style to Lombard style [13].…”

Multi-Metric Optimization Using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

“…Inspired by human speech production characteristics, some algorithms (e.g., [13], [14], [15]) aim to convert normal speech to Lombard speech [16], which is naturally produced by speakers with increased vocal effort for higher intelligibility. To achieve speaking style conversion, most algorithms rely on vocoder-based analysis-and-synthesis techniques, where vocoder features are transformed to fit in the Lombard style.…”

“…To achieve speaking style conversion, most algorithms rely on vocoder-based analysis-and-synthesis techniques, where vocoder features are transformed to fit in the Lombard style. For example, Seshadri et al [15] modified Mel-generalized cepstrum coefficients [17] of input speech to generate the Lombard-style speech by using log-domain pulse model vocoder [18]. However, using such a parametric vocoder inevitably degrades the converted speech quality.…”