Hand and Lip Desynchronization Analysis in French Cued Speech: Automatic Temporal Segmentation of Hand Flow

Aboutabit, Noureddine; Beautemps, Denis; Besacier, Laurent

doi:10.1109/icassp.2006.1660100

Cited by 12 publications

(16 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on previous studies, however, there might be asynchrony between the two components (Aboutabit et al, 2006). Late fusion (Potamianos et al, 2003), coupled HMMs (Nefian et al, 2002) and product HMMs (Nakamura et al, 2002) would be used as possible alternatives to the state-synchronous fusion methods used in this work.…”

Section: Discussionmentioning

confidence: 97%

Cued Speech automatic recognition in normal-hearing and deaf subjects

Heracleous¹,

Beautemps²,

Aboutabit³

2010

Speech Communication

View full text Add to dashboard Cite

This article discusses the automatic recognition of Cued Speech in French based on hidden Markov models (HMMs). Cued Speech is a visual mode which, by using hand shapes in different positions and in combination with lip patterns of speech, makes all the sounds of a spoken language clearly understandable to deaf people. The aim of Cued Speech is to overcome the problems of lipreading and thus enable deaf children and adults to understand spoken language completely. In the current study, the authors demonstrate that visible gestures are as discriminant as audible orofacial gestures. Phoneme recognition and isolated word recognition experiments have been conducted using data from a normal-hearing cuer. The results obtained were very promising, and the study has been extended by applying the proposed methods to a deaf cuer. The achieved results have not shown any significant differences compared to automatic Cued Speech recognition in a normal-hearing subject. In automatic recognition of Cued Speech, lip shape and gesture recognition are required. Moreover, the integration of the two modalities is of great importance. In this study, lip shape component is fused with hand component to realize Cued Speech recognition. Using concatenative feature fusion and multi-stream HMM decision fusion, vowel recognition, consonant recognition, and isolated word recognition experiments have been conducted. For vowel recognition, an 87.6% vowel accuracy was obtained showing a 61.3% relative improvement compared to the sole use of lip shape parameters. In the case of consonant recognition, a 78.9% accuracy was obtained showing a 56% relative improvement compared to the use of lip shape only. In addition to vowel and consonant recognition, a complete phoneme recognition experiment using concatenated feature vectors and Gaussian mixture model (GMM) discrimination was conducted, obtaining a 74.4% phoneme accuracy. Isolated word recognition experiments in both normalhearing and deaf subjects were also conducted providing a word accuracy of 94.9% and 89%, respectively. The obtained results were compared with those obtained using audio signal, and comparable accuracies were observed.

show abstract

Section: Discussionmentioning

confidence: 97%

Cued Speech automatic recognition in normal-hearing and deaf subjects

Heracleous¹,

Beautemps²,

Aboutabit³

2010

Speech Communication

View full text Add to dashboard Cite

show abstract

“…The temporal organization of hand movements in CS was studied in [15], [29], [30]. For CV syllables 2 , it was found that the hand position reaches its target before the vowel being T (L) , T (P) , T (S) are the temporal segmentations/boundaries for phonemes in case of lips, hand position and hand shape, respectively.…”

Section: B Multi-modal Temporal Modeling In Csmentioning

confidence: 99%

“…visible at lips on average 239ms [15] (based on the non-sense syllables logatome, like 'mamuma'), or 144.19ms [29] (based on the syllables extracted from French continuous sentences). In this work, we focus on not only the HPT of vowels, but also that of consonants.…”

Section: B Multi-modal Temporal Modeling In Csmentioning

confidence: 99%

Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

Liu

Feng

Beautemps

et al. 2021

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Cued Speech (CS) is an augmented lip reading system complemented by hand coding, and it is very helpful to the deaf people. Automatic CS recognition can help communications between the deaf people and others. Due to the asynchronous nature of lips and hand movements, fusion of them in automatic CS recognition is a challenging problem. In this work, we propose a novel re-synchronization procedure for multi-modal fusion, which aligns the hand features with lips feature. It is realized by delaying hand position and hand shape with their optimal hand preceding time which is derived by investigating the temporal organizations of hand position and hand shape movements in CS. This re-synchronization procedure is incorporated into a practical continuous CS recognition system that combines convolutional neural network (CNN) with multi-stream hidden markov model (MSHMM). A significant improvement of about 4.6% has been achieved retaining 76.6% CS phoneme recognition correctness compared with the state-of-the-art architecture (72.04%), which did not take into account the asynchrony issue of multi-modal fusion in CS. To our knowledge, this is the first work to tackle the asynchronous multi-modal fusion in the automatic continuous CS recognition.

show abstract

“…Based on that, it is worthy to add the changes (i.e., the slopes) in features like delta features and delta acceleration (delta‐delta) features. Moreover, it is possible to calculate the delta coefficients by using a linear regression formula considering the size of the regression window as 2C + 1 :

Δ I {normalC}_{l} = \frac{{true \sum}_{i = 1}^{C} i (I {normalC}_{l} true(m + i true) - I {normalC}_{l} true(m - i true)}{2 {true \sum}_{i = 1}^{C} i^{2}}

where

I {normalC}_{normall}

( m ) is the m th MFCC coefficient. The delta‐delta coefficients are found by utilizing linear regression of delta features.…”

Section: The Proposed Systemmentioning

confidence: 99%

An efficient speech recognition system for arm‐disabled students based on isolated words

Darabkh

Haddad

Sweidan

et al. 2017

Comp Applic In Engineering

View full text Add to dashboard Cite

Over the previous decades, a need has emerged to empower human‐machine communication systems, which are essential to not only perform actions, but also obtain information especially in education applications. Moreover, any communication system has to introduce an efficient and easy way for interaction with a minimum possible error rate. The keyboard, mouse, trackball, touch‐screen, and joystick are all examples of tools which were built to provide mechanical human‐to‐machine interaction. However, a system with the ability to use oral speech, which is the natural form of communication between humans instead of mechanical communication systems, can be more practical for normal students and even a necessity for arm‐disabled students who cannot use their arms to handle traditional education tools like pens and notebooks. In this paper, we present a speech recognition system that allows arm‐disabled students to control computers by voice as a helping tool in the educational process. When a student speaks through a microphone, the speech is divided into isolated words which are compared with a predefined database of huge number of spoken words to find a match. After that, each recognized word is translated into its related tasks which will be performed by the computer like opening a teaching application or renaming a file. The speech recognition process discussed in this paper involves two separate approaches; the first approach is based on double thresholds voice activity detection and improved Mel‐frequency cepstral coefficients (MFCC), while the second approach is based on discrete wavelet transform along with modified MFCC algorithm. Utilizing the best values for all parameters in just mentioned techniques, our proposed system achieved a recognition rate of 98.7% using the first approach, and 98.86% using the second approach of which is better in ratio than the first one but slower in processing which is a critical point for a real time system. Both proposed approaches were compared with other relevant approaches and their recognition rates were noticeably higher.

show abstract

Hand and Lip Desynchronization Analysis in French Cued Speech: Automatic Temporal Segmentation of Hand Flow

Cited by 12 publications

References 3 publications

Cued Speech automatic recognition in normal-hearing and deaf subjects

Cued Speech automatic recognition in normal-hearing and deaf subjects

Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition

An efficient speech recognition system for arm‐disabled students based on isolated words

Contact Info

Product

Resources

About