Ultrasound-Based Silent Speech Interface Built on a Continuous Vocoder

Abstract

Recently it was shown that within the Silent Speech Interface (SSI) field, the prediction of F0 is possible from Ultrasound Tongue Images (UTI) as the articulatory input, using Deep Neural Networks for articulatory-to-acoustic mapping. Moreover, text-to-speech synthesizers were shown to produce higher quality speech when using a continuous pitch estimate, which takes non-zero pitch values even when voicing is not present. Therefore, in this paper on UTI-based SSI, we use a simple continuous F0 tracker which does not apply a strict voiced / unvoiced decision. Continuous vocoder parameters (ContF0, Maximum Voiced Frequency and Mel-Generalized Cepstrum) are predicted using a convolutional neural network, with UTI as input. The results demonstrate that during the articulatory-toacoustic mapping experiments, the continuous F0 is predicted with lower error, and the continuous vocoder produces slightly more natural synthesized speech than the baseline vocoder using standard discontinuous F0.

Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review

“…Articulation is the ability to physically move the tongue, lips, teeth, and chin to produce a series of voice sounds that make up words and sentences. So an articulation-to-speech(ATS) synthesis is underway to restore the original voices of patients who produce abnormal languages, such as speech disorders [ 101 , 102 , 103 , 104 ]. Because this technique maps articulatory information directly to speech [ 104 ], it is possible to generate voice from articulatory movement data without subject making any sound [ 103 ].…”

“…So an articulation-to-speech(ATS) synthesis is underway to restore the original voices of patients who produce abnormal languages, such as speech disorders [ 101 , 102 , 103 , 104 ]. Because this technique maps articulatory information directly to speech [ 104 ], it is possible to generate voice from articulatory movement data without subject making any sound [ 103 ]. In addition to the sensors(EMA, PMA, sEMG) introduced in Section 2 , the movement of articulators can be captured using various information, such as ultrasound tongue imaging (UTI), non-audible murmur (NAM), and the advantage of being able to collect voice information, especially from people who can’t make sounds, enables compensation for insufficient training data [ 77 ].…”

“…So, using a large amount of motion data collected, people who have voice disorders can also synthesize with a natural voice. [ 102 , 103 , 104 ].…”

Silent Speech Interfaces for Speech Restoration: A Review

“…Prosody is mainly conditioned by the airflow and the vibration of the vocal folds, which in the case of laryngectomised patients is not possible to recover. As a result, most direct synthesis techniques generating a voice from sensed articulatory movements can, at best, recover a monotonous voice with limited pitch variations [101], [281], [282]. The use of complementary information capable of restoring prosodic features is thus an important area for future research.…”

Ultrasound-Based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis

“…The early studies on articulatory-to-acoustic mapping typically applied a low-order spectral representation, for example, only 12 coefficients were used in [8,10]. Later, our team also experimented with using 22 kHz speech and 24-order MGC-LSP target [15] (with the gain, having 25 dimensions altogether). Still, the 24-order MGC-LSP target is a relatively lowdimensional spectral representation, and this simple vocoder that we used in previous studies [10,11,12,13,14,15] can be a bottleneck in the ultrasound-to-speech mapping framework.…”