Estimating Articulatory Movements in Speech Production with Transformer Networks

Attia²,

Sivaraman³

et al. 2022

Preprint

Data augmentation has proven to be a promising prospect in improving the performance of deep learning models by adding variability to training data. In previous work with developing a noise robust acoustic-to-articulatory speech inversion system, we have shown the importance of noise augmentation to improve the performance of speech inversion in noisy speech. In this work, we compare and contrast different ways of doing data augmentation and show how this technique improves the performance of articulatory speech inversion not only on noisy speech, but also on clean speech data. We also propose a Bidirectional Gated Recurrent Neural Network as the speech inversion system instead of the previously used feed forward neural network. The inversion system uses mel-frequency cepstral coefficients (MFCCs) as the input acoustic features and six vocal tract-variables (TVs) as the output articulatory features. The Performance of the system was measured by computing the correlation between estimated and actual TVs on the U. Wisc. Xray Microbeam database. The proposed speech inversion system shows a 5% relative improvement in correlation over the baseline noise robust system for clean speech data. The pretrained model, when adapted to each unseen speaker in the test set, improves the average correlation by another 6%.

Section: Introductionmentioning

confidence: 99%

Audio Data Augmentation for Acoustic-to-articulatory Speech Inversion using Bidirectional Gated RNNs

Attia²,

Sivaraman³

et al. 2022

Preprint

“…Over the past few years, deep neural network (DNN) based models have propelled the development of SI systems to new heights. Bidirectional LSTMs (BiLSTMS) [15,16], CNN-BiLSTMs [17,18], Temporal Convolutional Networks (TCN) [19] and transformer models [20] have gained state-of-the-art results with multiple articulatory datasets [21]. To further improve the speech inversion task, people have tried incorporating phonetic transcriptions as an input along with acoustic features [22,17].…”

Section: Introductionmentioning

confidence: 99%

Acoustic-to-articulatory Speech Inversion with Multi-task Learning

Sivaraman²,

Espy‐Wilson³

2022

Preprint

Multi-task learning (MTL) frameworks have proven to be effective in diverse speech related tasks like automatic speech recognition (ASR) and speech emotion recognition. This paper proposes a MTL framework to perform acoustic-to-articulatory speech inversion by simultaneously learning an acoustic to phoneme mapping as a shared task. We use the Haskins Production Rate Comparison (HPRC) database which has both the electromagnetic articulography (EMA) data and the corresponding phonetic transcriptions. Performance of the system was measured by computing the correlation between estimated and actual tract variables (TVs) from the acoustic to articulatory speech inversion task. The proposed MTL based Bidirectional Gated Recurrent Neural Network (RNN) model learns to map the input acoustic features to nine TVs while outperforming the baseline model trained to perform only acoustic to articulatory inversion.

“…Recent advancements in deep neural networks (DNNs) and learning algorithms have significantly contributed to improving SI systems over the last few years. Bidirectional LSTMs (BiLSTMS) [14], BiGRNNs [15], CNN-BiLSTMs [16], Temporal Convolutional Networks (TCN) [17] and transformer models [18] have gained state-of-the-art results with multiple articulatory datasets. Most of these SI systems use either extracted acoustic features like Mel Frequency Cepstral Coefficients (MFCCs), Mel-spectrograms or the waveform itself as the input speech representation, and learns a mapping to the ground-truth articulatory variables.…”

Section: Introductionmentioning

confidence: 99%

The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversion

Espy‐Wilson²

2022

Preprint

In this work, we incorporated acoustically derived source features, aperiodicity, periodicity and pitch as additional targets to an acoustic-to-articulatory speech inversion (SI) system. We also propose a Temporal Convolution based SI system, which uses auditory spectrograms as the input speech representation, to learn long-range dependencies and complex interactions between the source and vocal tract, to improve the SI task. The experiments are conducted with both the Wisconsin X-ray microbeam (XRMB) and Haskins Production Rate Comparison (HPRC) datasets, with comparisons done with respect to three baseline SI model architectures. The proposed SI system with the HPRC dataset gains an improvement of close to 28% when the source features are used as additional targets. The same SI system outperforms the current best performing SI models by around 9% on the XRMB dataset.