Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling

Abstract

This study addresses the problem of unsupervised subword unit discovery from untranscribed speech. It forms the basis of the ultimate goal of ZeroSpeech 2019, building text-to-speech systems without text labels. In this work, unit discovery is formulated as a pipeline of phonetically discriminative feature learning and unit inference. One major difficulty in robust unsupervised feature learning is dealing with speaker variation. Here the robustness towards speaker variation is achieved by applying adversarial training and FHVAE based disentangled speech representation learning. A comparison of the two approaches as well as their combination is studied in a DNN-bottleneck feature (DNN-BNF) architecture. Experiments are conducted on ZeroSpeech 2019 and 2017. Experimental results on Ze-roSpeech 2017 show that both approaches are effective while the latter is more prominent, and that their combination brings further marginal improvement in across-speaker condition. Results on ZeroSpeech 2019 show that in the ABX discriminability task, our approaches significantly outperform the official baseline, and are competitive to or even outperform the official topline. The proposed unit sequence smoothing algorithm improves synthesis quality, at a cost of slight decrease in ABX discriminability.

Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling

“…The z 1 representation from a well-trained FHVAE is extracted as the desired speaker-invariant phonetic representation for unsupervised subword modeling. The FHVAE model was applied in [10] and achieved good performance in the ZeroSpeech 2019 Challenge [49], which is why we compare the APC model against FHVAE in this study. Details of the FHVAE model description is provided in supplementary material (see Section S1-A).…”

“…The phoneme-level analysis uses the 39 English phonemes in the CMU Dictionary [61]: these are 10 monophthongs, 5 diphthongs, and 24 consonants. Calculation of p co (ω i ) (see Equation (10)) depends on the ground-truth English phoneme labels and the cross-lingual phone labels. The English true phoneme labels for dev-clean are obtained by carrying out a forced alignment using the English TDNN AM that is described in Section V-B.…”

“…is a segment of a fixed number of frames. The FHVAE model formulates the generation process of a sequence X as 10 [23], 1) A vector µ 2 is drawn from a prior distribution p θ (µ 2 ) = N (0, σ 2 µ2 I); 2) Latent segment variables z n 1 and latent sequence variables z n 2 are drawn from…”

Tackling Perception Bias in Unsupervised Phoneme Discovery Using DPGMM-RNN Hybrid Model and Functional Load

“…Those clusters and speaker ids are trained with adversarial multi-task learning to get a final representation. The system has its primary representation (FHVAE(b)) and an alternative (FHVAE(a)) [58].…”

Bootstrapping Non-Parallel Voice Conversion from Speaker-Adaptive Text-to-Speech

“…Even though our scenario has the same objective as the challenge, the approach is a little different. The participant of the challenge are encouraged to develop intra-language unsupervised unit discovery methods, which are more difficult [32,33,34]. Our framework is bootstrapped from an abundant language, which is a more practical approach [35,36].…”