Cycle-consistent Adversarial Networks for Non-parallel Vocal Effort Based Speaking Style Conversion

Abstract

Speaking style conversion (SSC) is the technology of converting natural speech signals from one style to another. In this study, we propose the use of cycle-consistent adversarial networks (CycleGANs) for converting styles with varying vocal effort, and focus on conversion between normal and Lombard styles as a case study of this problem. We propose a parametric approach that uses the Pulse Model in Log domain (PML) vocoder to extract speech features. These features are mapped using the CycleGAN from utterances in the source style to the corresponding features of target speech. Finally, the mapped features are converted to a Lombard speech waveform with the PML. The CycleGAN was compared in subjective listening tests with 2 other standard mapping methods used in conversion, and the CycleGAN was found to have the best performance in terms of speech quality and in terms of the magnitude of the perceptual change between the two styles.

Augmented CycleGANs for Continuous Scale Normal-to-Lombard Speaking Style Conversion

“…The loss function of an augmented CycleGAN can be formulated similar to that of a standard CycleGAN used in [10]. In our implementation, we use the Wasserstein distance metric (WGAN loss) with gradient penalty [16] to determine the adverserial loss, defined as…”

“…where λcyc and λ id control the relative importance of the cyclic reconstruction loss and the identity mapping loss respectively. Following our previous works [9,10] on SSC, the current study uses a parametric system utilizing frame level Pulse Model in Log domain (PML [18]) vocoder features for the normal-to-Lombard conversion. The basic block diagram is shown in Figure 2.…”

“…In addition, a parametric approach to normal-to-Lombard SSC was recently explored in [9], where a vocoder was used to extract frame level features that were then transformed from normal to Lombard style using parallel data-driven mapping models, and then synthesized as speech in the target style using the same vocoder. The proposed vocoder-based SSC methodology was extended in [10] to a non-parallel learning scheme by using cycle consistent generative adversarial networks (Cy-cleGANs [11]), demonstrating superior quality and degree of Lombardness in comparison to the parallel-data system in [9].…”

“…Given this background, the goal of the current study is to explore the applicability of Augmented CycleGANs in a normal-to-Lombard SSC with the aim of having controllable degree of Lombardness in the conversion system. The paper builds on the data driven approach to the normal-to-Lombard parametric SSC system explored in [8][9][10] by using the Pulse model in log domain (PML [11]) vocoder with the Augmented CycleGANs [2]. We compare the new system to the reference CycleGAN system proposed for the same task in [10]…”

Novel Inception-GAN for Whispered-to-Normal Speech Conversion

“…Right after the GAN was invented [16], many GAN-based architectures actively used for VC [7,[17][18][19][20][21][22]. Recently, Cycleconsistent Adversarial Network (CycleGAN) gives the state-ofthe-art results in VC [23][24][25][26]. While other variants of GANs, such as MMSE DiscoGAN proposed in [27] is also shown significant results for WHSP2SPCH conversion task, in particular, for prediction of F0.…”