Lips and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications

Cosi, Piero; Caldognetto, Emanuela Magno

doi:10.1007/978-3-662-13015-5_23

Cited by 13 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Seventeen valid sequences of all vowels were obtained in the end. This sample size is comparable to that of other studies [3, 4, 6, 7, 9, 10, 13]. …”

Section: Data Collection and Processingsupporting

confidence: 85%

Vowel recognition from articulatory position time-series data

Wang

Samal

Green

et al. 2009

2009 3rd International Conference on Signal Processing and Communication Systems

View full text Add to dashboard Cite

Abstract--A new approach of recognizing vowels from articulatory position time-series data was proposed and tested in this paper. This approach directly mapped articulatory position time-series data to vowels without extracting articulatory features such as mouth opening. The input time-series data were time-normalized and sampled to fixed-width vectors of articulatory positions. Three commonly used classifiers, Neural Network, Support Vector Machine and Decision Tree were used and their performances were compared on the vectors. A single speaker dataset of eight major English vowels acquired using Electromagnetic Articulograph (EMA) AG500 was used. Recognition rate using cross validation ranged from 76.07% to 91.32% for the three classifiers. In addition, the trained decision trees were consistent with articulatory features commonly used to descriptively distinguish vowels in classical phonetics. The findings are intended to improve the accuracy and response time of a real-time articulatory-to-acoustics synthesizer.

show abstract

“…Seventeen valid sequences of all vowels were obtained in the end. This sample size is comparable to that of other studies [3, 4, 6, 7, 9, 10, 13]. …”

Section: Data Collection and Processingsupporting

confidence: 85%

Vowel recognition from articulatory position time-series data

Wang

Samal

Green

et al. 2009

2009 3rd International Conference on Signal Processing and Communication Systems

View full text Add to dashboard Cite

show abstract

“…We focused on bilabial consonants because our X-ray database provided only a one-dimensional measure of mouth opening. Some prior work has attempted to measure the range of lip and jaw movements for a larger range of vowels and consonants, but only with a few subjects; estimates from this study were similar to ours [78]. Future work using the methods outlined in [78] or using the chroma key system [22] could be used to measure the time-to-voice for a larger range of articulatory contexts.…”

Section: Discussionmentioning

confidence: 53%

The Natural Statistics of Audiovisual Speech

et al. 2009

View full text Add to dashboard Cite

Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2–7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver.

show abstract

“…This approach is based on the assumption that a small set of articulatory features can be used to distinguish vowels. These features include lip rounding/lip opening, tongue tip position [12], [13], [14], [16], lip contour or area [8], [15], [16], visemes [17], vertical and horizontal lip apertures, angles of lips [18], lip opening height and width, velocity of lip opening/closing, acceleration of lip movement [19]. However, seldom have these features resulted in a recognition accuracy greater than 90%.…”

Section: Introductionmentioning

confidence: 99%

Vowel recognition from continuous articulatory movements for speaker-dependent applications

Wang

Green

Samal

et al. 2010

2010 4th International Conference on Signal Processing and Communication Systems

View full text Add to dashboard Cite

Abstract-A novel approach was developed to recognize vowels from continuous tongue and lip movements. Vowels were classified based on movement patterns (rather than on derived articulatory features, e.g., lip opening) using a machine learning approach. Recognition accuracy on a single-speaker dataset was 94.02% with a very short latency. Recognition accuracy was better for high vowels than for low vowels. This finding parallels previous empirical findings on tongue movements during vowels. The recognition algorithm was then used to drive an articulation-to-acoustics synthesizer. The synthesizer recognizes vowels from continuous input stream of tongue and lip movements and plays the corresponding sound samples in near real-time.

show abstract

Lips and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications

Cited by 13 publications

References 16 publications

Vowel recognition from articulatory position time-series data

Vowel recognition from articulatory position time-series data

The Natural Statistics of Audiovisual Speech

Vowel recognition from continuous articulatory movements for speaker-dependent applications

Contact Info

Product

Resources

About