Toward invariant functional representations of variable surface fundamental frequency contours: Synthesizing speech melody via model-based stochastic learning

Abstract

Variability has been one of the major challenges for both theoretical understanding and computer synthesis of speech prosody. In this paper we show that economical representation of variability is the key to effective modeling of prosody. Specifically, we report the development of PENTAtrainer -A trainable yet deterministic prosody synthesizer based on an articulatory-functional view of speech. We show with testing results on Thai, Mandarin and English that it is possible to achieve high-accuracy predictive synthesis of fundamental frequency contours with very small sets of parameters obtained through stochastic learning from real speech data. The first key component of this system is syllable-synchronized sequential target approximation -implemented as the qTA model, which is designed to simulate, for each tonal unit, a wide range of contextual variability with a single invariant target. The second key component is the automatic learning of function-specific targets through stochastic global optimization, guided by a layered pseudo-hierarchical functional annotation scheme, which requires the manual labeling of only the temporal domains of the functional units. The results in terms of synthesis accuracy demonstrate that effective modeling of the contextual variability is the key also to effective modeling of function-related variability. Additionally, we show that, being both theory-based and trainable (hence datadriven), computational systems like PENTAtrainer can serve as an effective modeling tool in basic research, with which the level of falsifiability in theory testing can be raised, and also a closer link between basic and applied research in speech science can be developed.3 Graphical Abstract Highlights (maximum 85 characters/bullet)• High synthetic accuracy of prosody achieved for Thai, Mandarin and English • Many-to-one mapping from contextually variable surface F 0 to invariant functional targets • Effectively handling of both contextual and non-contextual variability • Combination of deterministic synthesis and data-driven parameter learning • Large-scale and full-detailed prosody synthesis as tool for theory testing • Freely available as a Praat scripts and plug-ins to the speech science community at large

Fronted NPs in a verb-initial language – clause-internal or external? Prosodic cues to the rescue!

“…For the purpose of this analysis, only tokens evaluated on a perceptual basis as being uttered with a "normal" rate were selected, and no normalization was applied. The PRAAT scripts ProsodyPro (Xu & Prom-on 2014), and PENTAtrainer 1 (Prom-on, Xu & Thipakorn 2009) were then used to obtain the measurements of the prosodic correlates that include duration (in milliseconds), intensity (in decibels), mean F0 (average of 10 measurements over the syllable, in Hz), and excursion size (F0maxima -minima, in semitones).…”

Predicting tonal realizations in one Chinese dialect from another

“…Although the majority of modern Chinese words are disyllabic and the uncertainty of disyllabic tonal realizations has partly been explained by individual backgrounds, how disyllabic JM words are realized in connected speech needs further investigation. Researchers have done fruitful studies on contextual tonal realizations and their interactions with sentential prosodies (Chen, 2010;Chen and Gussenhoven, 2008;Xu, 1997Xu, , 1999Xu and Prom-on, 2014;Xu and Wang, 2001). However, how to transfer this knowledge from SC to the other Chinese dialects and how the predictors we investigated work in context still open questions.…”

The Common Prosody Platform (CPP): Where theories of prosody can be directly compared

“…PENTA has been implemented to perform both local and global optimization methods [2,9]. The detailed implementation of PENTA with global optimization is given in [9]. Target approximation (TA) in PENTA is mathematically realized as a third-order critically damped linear system driven by pitch targets, as shown in:…”

“…To test the CPP program, we took subsets of two corpora used previously in the development of PENTAtrainer2 [9]. The first corpus was collected for a study of tone, focus and sentence modality in Mandarin Chinese and the second one was collected for a study of stress, focus and sentence modality in American English [12].…”

“…In terms of learning algorithms, at the present stage, all four models have been implemented with local curve fitting capabilities, while only PENTA has been implemented with full-fledged global optimization algorithms in PENTAtrainer2 [9]. So this paper will focus mainly on the local fitting capabilities of all models, the software package that provides a means of comparing all models simultaneously and the results of testing them on Mandarin and English test corpora.…”