Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips

Abstract

This article presents a segmental vocoder driven by ultrasound and optical images (standard CCD camera) of the tongue and lips for a "silent speech interface" application, usable either by a laryngectomized patient or for silent communication. The system is built around an audiovisual dictionary which associates visual to acoustic observations for each phonetic class. Visual features are extracted from ultrasound images of the tongue and from video images of the lips using a PCA-based image coding technique. Visual observations of each phonetic class are modeled by continuous HMMs. The system then combines a phone recognition stage with corpus-based synthesis. In the recognition stage, the visual HMMs are used to identify phonetic targets in a sequence of visual features. In the synthesis stage, these phonetic targets constrain the dictionary search for the sequence of diphones that maximizes similarity to the input test data in the visual space, subject to a concatenation cost in the acoustic domain. A prosody template is extracted from the training corpus, and the final speech waveform is generated using "Harmonic plus Noise Model" concatenative synthesis techniques. Experimental results are based on an audiovisual database containing one hour of continuous speech from each of two speakers.

Automatic speech recognition using acoustic doppler signal

“…무음 성 전달 방법으로서, 입주변에서 발생하는 근전도 신호를 이용하는 방법, [2] NAM(Non-Audible Microphone) 을 입주변에 부착하여 음성을 취득하는 방법, [3] 자석 과 자계센서를 이용하는 방법, [4] 구강 및 비강의 초음 파 영상을 이용한 방법, [5] GHz microwave를 이용하는 방법, [6] 초음파 신호를 이용하는 방법 [7] 등을 들 수 있다. Kalgaonkar et al [8] 의 연구에서는 간단한 손동작을 초음파 도플러를 이용하여 인식하였을 때 평균 88.4 % 의 인식율 [10] 이, 보행 패턴을 인식하는 경우 91.7 %의 인식율을 얻는 것으로 보고하였다.…”

Voice Restoration After Laryngectomy Based on Magnetic Sensing of Articulator Movement and Statistical Articulation-to-Speech Conversion

“…Many different SSIs have been proposed so far, mainly differing in the type of biosignal they rely on. Thus, we can find SSIs that exploit the electrical signals generated by the neurons in the brain [23] or in the articulator muscles [31,42,49] or the movement of the speech articulators themselves [40,44,9,29,18,14,26,21]. In our work we use a magnetic sensing technique known as Permanent Magnet Articulography (PMA) [13,18] for capturing the movement of the speech articulators.…”

Proceedings of the 5th Workshop on Speech and Language Processing for Assistive Technologies

“…Although still in developmental stages (e.g., speakerdependent recognition, small-vocabulary), SSIs even have potential to provide speech output based on prerecorded samples of the patient's own voice Green et al, 2011;Wang et al, 2009). Potential articulatory data acquisition methods for SSIs include ultrasound (Denby et al, 2011;Hueber et al, 2010), surface electromyography electrodes (Heaton et al, 2011;Jorgensen and Dusan, 2010), and electromagnetic articulograph (EMA) (Fagan et al, 2008;Wang et al, 2009Wang et al, , 2012a.…”

“…So far, most of the published work on SSIs has focused on development of silent speech recognition algorithm through offline analysis (i.e., using prerecorded data) (Fagan et al, 2008;Heaton et al, 2011;Hofe et al, 2013;Hueber et al, 2010;Jorgenson et al, 2010;Wang et al, 2009aWang et al, , 2012aWang et al, , 2012bWang et al, , 2013c. Ultrasoundbased SSIs have been tested online with multiple subjects and encouraging results were obtained in a phrase reading task where the subjects were asked to silently articulate sixty phrases (Denby et al, 2011).…”