Evaluating Long-form Text-to-Speech: Comparing the Ratings of Sentences and Paragraphs

Abstract

Text-to-speech systems are typically evaluated on single sentences. When long-form content, such as data consisting of full paragraphs or dialogues is considered, evaluating sentences in isolation is not always appropriate as the context in which the sentences are synthesized is missing.In this paper, we investigate three different ways of evaluating the naturalness of long-form text-to-speech synthesis. We compare the results obtained from evaluating sentences in isolation, evaluating whole paragraphs of speech, and presenting a selection of speech or text as context and evaluating the subsequent speech. We find that, even though these three evaluations are based upon the same material, the outcomes differ per setting, and moreover that these outcomes do not necessarily correlate with each other. We show that our findings are consistent between a single speaker setting of read paragraphs and a two-speaker dialogue scenario. We conclude that to evaluate the quality of long-form speech, the traditional way of evaluating sentences in isolation does not suffice, and that multiple evaluations are required.

Camp: A Two-Stage Approach to Modelling Prosody in Context

“…Since we provide isolated sentences to listeners, there is no concept of correct prosody [5], as such we do not provide a visible reference and do not require listeners to rate two systems as 0 and 100. 25 listeners completed this test.…”

“…Unrealistic prosody modelling is often linked to a lack of contextual information. Indeed, without sufficient context, predicting prosody is an ill-posed problem [5], as any number of prosodies could be deemed appropriate for a given text. We believe that this is the biggest limitation of current state-of-the-art TTS models.…”

Breathing and Speech Planning in Spontaneous Speech Synthesis

“…Despite the interest in TTS that speaks longer chunks of text, e.g., [16], no major engine currently assumes a connection between consecutive audio files in the corpus. Instead, utterances are traditionally treated in isolation.…”

“…The main reasons for retaining this custom is that 1) most conventional TTS training material is comprised of sentences read in isolation, and 2) loading longer speech segments (e.g., audiobook paragraphs instead of sentences) into a conventional TTS engine would quickly result in the system running out of memory. This state of affairs is unfortunate in the case of corpora created from longer continuous recordings of speech, such as audiobooks [16,17]. It is even more problematic for spontaneous speech, where syntactic forms do not follow the conventions of written language and cannot be easily separated into standalone, coherent semantic units.…”

Prosodic Prominence and Boundaries in Sequence-to-Sequence Speech Synthesis

“…Another, somewhat complementary reason arises from the lack of explicit control inherent to the "black-box" machine learning architectures, such as s2s systems. On the one hand, the existing systems are not designed to capture the long-range semantic dependencies [7], on the other hand, they do not facilitate explicit control of prosody akin to older parametric synthesis approaches, where linguistic and prosodic labels were utilized and prosodic parameters were modelled separately [8].…”