Acoustic-to-articulatory Speech Inversion with Multi-task Learning

Siriwardena, Yashish M.; Sivaraman, Ganesh; Espy‐Wilson, Carol Y.

doi:10.21437/interspeech.2022-11164

Cited by 5 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Hprc Ema Datasetmentioning

confidence: 99%

“…For our AAI train-val-test split, we hold out 1 male and 1 female speaker for our test set and train on the remaining 6 speakers, as done in [15]. While [15] also used these two speakers in their validation set, we put all of the data from both speakers in the test set in order to fully experiment within the unseen speaker setting. We note that our formulation increases the difficulty of the task compared to [15] since hyperparameter tuning would not lead to overfitting on the test speakers.…”

Section: Hprc Ema Datasetmentioning

confidence: 99%

“…While [15] also used these two speakers in their validation set, we put all of the data from both speakers in the test set in order to fully experiment within the unseen speaker setting. We note that our formulation increases the difficulty of the task compared to [15] since hyperparameter tuning would not lead to overfitting on the test speakers. However, our AAI method still obtains state-of-theart Pearson correlation scores with this dataset, as detailed in Section 5.…”

Section: Hprc Ema Datasetmentioning

confidence: 99%

“…For our baseline AAI model, we use the bidirectional-GRUbased architecture in [15]. Specifically, this model is comprised of a two-layer bidirectional GRU [33] with size-256 hidden states followed by a two-layer multi-layer perceptron (MLP) [34] with 128 hidden units.…”

Section: Articulatory Inversion Modelmentioning

confidence: 99%

“…In recent years, deep learning algorithms have become popular state-of-the-art methods for AAI [15,16,17,18,19, Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Speaker-Independent Acoustic-to-Articulatory Speech Inversion

Wu¹,

Chen²,

Cho³

et al. 2023

Preprint

View full text Add to dashboard Cite

To build speech processing methods that can handle speech as naturally as humans, researchers have explored multiple ways of building an invertible mapping from speech to an interpretable space. The articulatory space is a promising inversion target, since this space captures the mechanics of speech production. To this end, we build an acoustic-toarticulatory inversion (AAI) model that leverages autoregression, adversarial training, and self supervision to generalize to unseen speakers. Our approach obtains 0.784 correlation on an electromagnetic articulography (EMA) dataset, improving the state-of-the-art by 12.5%. Additionally, we show the interpretability of these representations through directly comparing the behavior of estimated representations with speech production behavior. Finally, we propose a resynthesis-based AAI evaluation metric that does not rely on articulatory labels, demonstrating its efficacy with an 18-speaker dataset.

show abstract

Section: Hprc Ema Datasetmentioning

confidence: 99%