Voice Filter: Few-Shot Text-to-Speech Speaker Adaptation Using Voice Conversion as a Post-Processing Module

Gabryś, Adam; Huybrechts, Goeric; Ribeiro, Manuel Sam; Chien, Chung-Ming; Roth, Julian; Comini, Giulia; Barra-Chicote, Roberto; Perz, Bartek; Lorenzo-Trueba, Jaime

doi:10.1109/icassp43922.2022.9747239

Cited by 10 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each of these recordings are paired with a synthetically generated audio of a target speaker, to train the manyto-one voice conversion model. These syntheses have been generated using a similar technique of the one described in this paper [9]. As mentioned in the section 4.6 of the manuscript, the evaluation set is composed of 3500 utterances, which were held out from the training set.…”

Section: Downstream Application 2: Text-less Intelligibility Evaluati...mentioning

confidence: 99%

SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces

Valles-Perez,

Beringer,

Bilinski

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Numerous examples in the literature proved that deep learning models have the ability to work well with multimodal data. Recently, CLIP has enabled deep learning systems to learn shared latent spaces between images and text descriptions, with outstanding zero- or few-shot results in downstream tasks. In this paper we explore the same idea proposed by CLIP but applied to the speech domain, where the phonetic and acoustic spaces usually coexist. We train a CLIP-based model with the aim to learn shared representations of phonetic and acoustic spaces. The results show that the proposed model is sensible to phonetic changes, with a 91% of score drops when replacing 20% of the phonemes at random, while providing substantial robustness against different kinds of noise, with a 10% performance drop when mixing the audio with 75% of Gaussian noise. We also provide empirical evidence showing that the resulting embeddings are useful for a variety of downstream applications, such as intelligibility evaluation and the ability to leverage rich pre-trained phonetic embeddings in speech generation task. Finally, we discuss potential applications with interesting implications for the speech generation and recognition fields.

show abstract

Section: Downstream Application 2: Text-less Intelligibility Evaluati...mentioning

confidence: 99%

SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces

Valles-Perez,

Beringer,

Bilinski

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…For objective evaluation, we first calculate the average Speaker Embedding Cosine Similarity (SECS) between the reference and measured audios by a speaker verification model [24] to estimate speaker similarity. Further, we compute Conditional Fréchet Speech Distance (CFSD) [14] between the generated speech and actual recording to measure signal quality. Besides, we also evaluate mean square error for pitch (MSE P ) and duration (MSE D ) to access prosody similarity.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…lated to speaker identity [10,11,12,13]. The other alternative approach is based on using a light voice conversion postprocessing module to baseline TTS model [14]. The third challenge is to reduce amount of speech required to add new speaker to existing TTS model.…”

Section: Introductionmentioning

confidence: 99%

Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers

Ghosh¹,

Ginsburg²

2022

Preprint

View full text Add to dashboard Cite

Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However this approach has some challenges. Usually fine-tuning requires several hours of high quality speech per speaker. There is also that finetuning will negatively affect the quality of speech synthesis for previously learnt speakers. In this paper we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few small adapter modules are added to the original network. The original weights are frozen, and only the adapters are fine-tuned on speech for new speaker. The parameter-efficient fine-tuning approach will produce a new model with high level of parameter sharing with original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the effectiveness of adapter-based method through objective and subjective metrics.

show abstract