Towards the Prediction of the Vocal Tract Shape from the Sequence of Phonemes to be Articulated

Abstract

In this work, we address the prediction of speech articulators' temporal geometric position from the sequence of phonemes to be articulated. We start from a set of real-time MRI sequences uttered by a female French speaker. The contours of five articulators were tracked automatically in each of the frames in the MRI video. Then, we explore the capacity of a bidirectional GRU to correctly predict each articulator's shape and position given the sequence of phonemes and their duration. We propose a 5-fold cross-validation experiment to evaluate the generalization capacity of the model. In a second experiment, we evaluate our model's data efficiency by reducing training data. We evaluate the point-to-point Euclidean distance and the Pearson's correlations along time between the predicted and the target shapes. We also evaluate produced shapes of the critical articulators of specific phonemes. We show that our model can achieve good results with minimal data, producing very realistic vocal tract shapes.

Autoencoder-Based Tongue Shape Estimation During Continuous Speech

“…A probable explanation for this effect is that the ground truth is noisy once it is subjected to tracking errors. These tracking errors in the target curve impose a performance upper bound in the previous approach [8]. However, since we enforce phoneme-wise constraints in the reconstruction, the critical loss inputs prior domain knowledge to the model generating a potentially more realistic result than the ground truth, which explains why the ρTBCD and ρTTCD are slightly lower than in the previous work.…”

“…The encoder-decoder network that maps phonemes to the autoencoder's latent space is very similar to the one used in [8]. The same GRU-based encoder with a linear reshaping layer is used.…”

“…Previous work [8,9] proposed an encoder-decoder neural network to predict the vocal tract shape for a sequence of phonemes to be articulated. The former was the first to provide the complete vocal tract shape, including all speech articulators from the glottis to the lips.…”

Automatic generation of the complete vocal tract shape from the sequence of phonemes to be articulated

“…Along with this work, we extend the research to a larger dataset than that used in [23]. The dataset [24] is composed of one male French native speaker.…”

“…In our most recent work, we proposed the first attempt to predict the vocal tract shape from the phonemes to be articulated [23]. We proposed a deep neural network to predict the positions of five articulators, i.e., the tongue, the upper and lower lips, the soft palate, and the pharyngeal wall.…”