How to compare TTS systems: a new subjective evaluation methodology focused on differences

Chevelu, Jonathan; Lolive, Damien; Maguer, Sébastien Le; Guennec, David

doi:10.21437/interspeech.2015-690

Cited by 8 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pairwise comparison tests have been shown to mitigate this. The testing material can also be chosen to better highlight differences between systems, thereby making listening tests more efficient [153], a task that some have suggested can be facilitated by automatic quality predictors as well [74].…”

Section: Future Prospects and Challengesmentioning

confidence: 99%

A review on subjective and objective evaluation of synthetic speech

Cooper,

Huang,

Tsao

et al. 2024

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

Evaluating synthetic speech generated by machines is a complicated process, as it involves judging along multiple dimensions including naturalness, intelligibility, and whether the intended purpose is fulfilled. While subjective listening tests conducted with human participants have been the gold standard for synthetic speech evaluation, its costly process design has also motivated the development of automated objective evaluation protocols. In this review, we first provide a historical view of listening test methodologies, from early in-lab comprehension tests to recent large-scale crowdsourcing mean opinion score (MOS) tests. We then recap the development of automatic measures, ranging from signal-based metrics to model-based approaches that utilize deep neural networks or even the latest self-supervised learning techniques. We also describe the VoiceMOS Challenge series, a scientific event we founded that aims to promote the development of data-driven synthetic speech evaluation. Finally, we provide insights into unsolved issues in this field as well as future prospective. This review is expected to serve as an entry point for early academic researchers to enrich their knowledge in this field, as well as speech synthesis practitioners to catch up on the latest developments.

show abstract

Section: Future Prospects and Challengesmentioning

confidence: 99%

A review on subjective and objective evaluation of synthetic speech

Cooper,

Huang,

Tsao

et al. 2024

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

show abstract

“…We note that the L 1 loss we use is comparatively simple, as spectrogram losses in the literature tend to have separate terms penalising magnitudes, log-magnitudes and phase components, each with their own scaling factors, and often across multiple resolutions. Dynamic time warping on spectrograms is a component of many speech recognition systems [52,53], and has also been used for evaluation of TTS systems [10,51]. Cuturi and Blondel [12] proposed the soft version of DTW we use in this work as a differentiable loss function for time series models.…”

Section: Related Workmentioning

confidence: 99%

End-to-End Adversarial Text-to-Speech

Donahue¹,

Dieleman²,

Bińkowski³

et al. 2020

Preprint

View full text Add to dashboard Cite

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable monotonic interpolation scheme to predict the duration of each input token. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision. 1

show abstract