A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages

Do, Phat; Coler, Matt; Dijkstra, Jelske; Klabbers, Esther

doi:10.21437/interspeech.2021-1565

Cited by 8 publications

(4 citation statements)

References 43 publications

(104 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, as mentioned in § IV-A, we choose Spanish and French if the target language is Italian. It should be noted that while recent work [42] has proposed a metric for TTS transfer learning, there is no universal metric to measure language similarity. In this study, we select the similar languages for the cross-lingual transfer based on language family defined in Glottolog [43].…”

Section: B Text-based Adaptationmentioning

confidence: 99%

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Saeki,

Maiti,

et al. 2024

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Neural text-to-speech (TTS) systems have made significant progress in generating natural synthetic speech. However, neural TTS requires large amounts of paired training data, which limits its applicability to a small number of resource-rich languages. Previous work on low-resource TTS has addressed the data hungriness based on transfer learning from a multilingual model to low-resource languages, but it still relies heavily on the availability of paired data for the target languages. In this paper, we propose a text-inductive language adaptation framework for low-resource TTS to address the cost of collecting the paired data for low-resource languages. To inject textual knowledge during transfer learning, our framework employs a two-stage adaptation scheme that utilizes both text-only and supervised data for the target language. In the text-based adaptation stage, we update the language-aware embedding layer with a masked language model objective using text-only data for the target language. In the supervised adaptation stage, the entire TTS model is updated using paired data for the target language. We also propose a graphone-based multilingual training method that jointly uses graphemes and International Phonetic Alphabet symbols (referred to as graphones) for resource-rich languages, while using only graphemes for low-resource languages. This approach facilitates the transfer of pronunciation knowledge from resource-rich to low-resource languages. Through extensive evaluations, we demonstrate that 1) our framework with textbased adaptation outperforms the previous supervised transfer learning approach, 2) the proposed graphone-based training method further improves the performance of both multilingual TTS and low-resource language adaptation. With only 5 minutes of paired data for fine-tuning, our method achieved highly intelligible synthetic speech with the character error rates of around 6 % for a target language.

show abstract

Section: B Text-based Adaptationmentioning

confidence: 99%

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Saeki,

Maiti,

et al. 2024

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Although the TTS systems in [5] are trained based on language families, the relevance of training them in this manner is not explored. A recent study [60] observes that language family classification may not be an effective basis for choosing (source) languages for training a generic TTS. However, our own experiences with multilingual TTS systems and observations in [18] find that the intelligibility of synthesised speech depends on the similarity between any target language and source language(s).…”

Section: Related Workmentioning

confidence: 99%

Exploring the Role of Language Families for Building Indic Speech Synthesisers

Prakash

Murthy

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Building end-to-end speech synthesisers for Indian languages is challenging, given the lack of adequate clean training data and multiple grapheme representations across languages. This work explores the importance of training multilingual and multi-speaker text-to-speech (TTS) systems based on language families. The objective is to exploit the phonotactic properties of language families, where small amounts of accurately transcribed data across languages can be pooled together to train TTS systems. These systems can then be adapted to new languages belonging to the same family in extremely low-resource scenarios.TTS systems are trained separately for Indo-Aryan and Dravidian language families, and their performance is compared to that of a combined Indo-Aryan+Dravidian voice. We also investigate the amount of training data required for a language in a multilingual setting. Same-family and cross-family synthesis and adaptation to unseen languages are analysed. The analyses show that language family-wise training of Indic systems is the way forward for the Indian subcontinent, where a large number of languages are spoken.Index Terms-end-to-end speech synthesis, Indian languages, language families, low-resource• This work is one of the first attempts to study the importance of language families in the context of speech synthesis.• We compare language family-specific Indo-Aryan (IA) and Dravidian (Dr) models with a combined Indo-Aryan+Dravidian (IA+Dr) system. • We also assess the performance of models trained in datastressed situations. We reduce the training data used per language in the multilingual voice.

show abstract

“…Objective metrics listed in Table 2 are divided into two aspects a: a) robustness: it quantifies how synthesized speech is correctly transcribed by listeners, such as character error rate (CER), word error rate (WER) and word information lost (WIL) [18,19]; and b) latency: it measures the latency to synthesize speech, such as the real time factor (RTF) in [12].…”

Section: Objective Metricsmentioning

confidence: 99%

Performance Comparison of TTS Models for Brazilian Portuguese to Establish a Baseline

Lobato¹,

Farias²,

Castañeda³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper compares the performance of three text-to-speech (TTS) models released from June 2021 to January 2022 in order to establish a baseline for Brazilian Portuguese. Those models were trained using dataset for Brazilian Portuguese. The experimental setup considers tts-portuguese dataset to fine-tune the following TTS models: VITS end-to-end model; glowtts and gradtts acoustic models both using hifi-gan vocoder. Performance metrics are arranged into objective and subjective metrics. As subjective metrics, the naturalness and intelligibility are measured based on the mean opinion score (MOS). Results shows that gradtts+hifigan model achieved naturalness of 4.07 MOS, close to performance of current commercial models.

show abstract

A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages

Cited by 8 publications

References 43 publications

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Exploring the Role of Language Families for Building Indic Speech Synthesisers

Performance Comparison of TTS Models for Brazilian Portuguese to Establish a Baseline

Contact Info

Product

Resources

About