Audio-Visual Speech Recognition Using Bimodal-Trained Bottleneck Features for a Person with Severe Hearing Loss

Takashima, Yuki; Aihara, Ryo; Takiguchi, Tetsuya; Ariki, Yasuo; Mitani, Nobuyuki; Omori, Kiyohiro; Nakazono, Kaoru

doi:10.21437/interspeech.2016-721

Cited by 22 publications

(15 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both voice and visual data were used in the method and extracted features were included in system for classification. In another work, Takashima et al [24] proposed a new approach for lip reading using the combination of lip image and sound features by using deep learning. The proposed method was tested on ATR Japanese speech dataset.…”

Section: Related Workmentioning

confidence: 99%

Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models

Ozcan

Baştürk

2019

Balkan Journal of Electrical and Computer Engineering

View full text Add to dashboard Cite

Lip reading has become a popular topic recently. There are widespread literature studies on lip reading in human action recognition. Deep learning methods are frequently used in this area. In this paper, lip reading from video data is performed using self designed convolutional neural networks (CNNs). For this purpose, standard and also augmented AvLetters dataset is used in train and test stages. To optimize network performance, minibatchsize parameter is also tuned and its effect is investigated. Additionally, experimental studies are performed using AlexNet and GoogleNet pre-trained CNNs. Detailed experimental results are presented.

show abstract

Section: Related Workmentioning

confidence: 99%

Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models

Ozcan

Baştürk

2019

Balkan Journal of Electrical and Computer Engineering

View full text Add to dashboard Cite

show abstract

“…Recently in [3] automatic classification using Support Vector Machines (SVM) between 20 CI users and 20 healthy speakers was performed in order to evaluate articulation disorders considering acoustic features. For the case of pathological speech detection, CNNs have outperformed classical machine learning methods [4][5][6]. In these studies, the conventional method is to perform time-frequency analysis by computing spectrograms over the speech signals to feed the CNNs with single channel inputs.…”

Section: Introductionmentioning

confidence: 99%

Multi-channel Convolutional Neural Networks for Automatic Detection of Speech Deficits in Cochlear Implant Users

Arias-Vergara

Vásquez-Correa

Gollwitzer

et al. 2019

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications

View full text Add to dashboard Cite

This paper proposes a methodology for automatic detection of speech disorders in Cochlear Implant users by implementing a multi-channel Convolutional Neural Network. The model is fed with a 2-channel input which consists of two spectrograms computed from the speech signals using Mel-scaled and Gammatone filter banks. Speech recordings of 107 cochlear implant users (aged between 18 and 89 years old) and 94 healthy controls (aged between 20 and 64 years old) are considered for the tests. According to the results, using 2-channel spectrograms improves the performance of the classifier for automatic detection of speech impairments in Cochlear Implant users.

show abstract

“…For these reasons, when targeting continuous lip-reading it is convenient to predict 1340 smaller structures that approach the minimum distinguishable language units. Recent advances in end-toend DL architectures have indeed focused on ALR systems that try to predict phonemes [149,51,139,83] or characters [16,25,161,165], instead of full words 1345 or pre-defined sentences. For example, Mroueh et al [83] proposed Feed-forward DNNs to predict phonemes using the IBM AV-ASR database, a large scale nonpublic AV database.…”

mentioning

confidence: 99%

“…For example, Mroueh et al [83] proposed Feed-forward DNNs to predict phonemes using the IBM AV-ASR database, a large scale nonpublic AV database. Other architectures using CNNs and HMMs were presented by Noda et al [51,139] and by Takashima et al [149]. They tried to recognize Japanese phonemes using the ATR Japanese corpus [178] and obtained 22.50% WRR, 37.00% WRR and 51.00% WRR, respectively.…”

mentioning

confidence: 99%

“…Thus, recent attempts to produce continuous lip-reading systems have focused on elementary language structures such as characters or phonemes. For instance, hydrid architectures for continuous speech recognition in Japanese [51,139,149] have targeted phonemes achieving between 22% and 51% WRR, while Chung et al [16] and Afouras et al [161] achieved near 50% WRR targeting characters for a large-scale dataset in English.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Survey on automatic lip-reading in the era of deep learning

Fernandez-Lopez

Sukno

2018

Image and Vision Computing

View full text Add to dashboard Cite

In the last few years, there has been an increasing interest in developing systems for Automatic LipReading (ALR). Similarly to other computer vision applications, methods based on Deep Learning (DL) have become very popular and have permitted to substantially push forward the achievable performance. In this survey, we review ALR research during the last decade, highlighting the progression from approaches previous to DL (which we refer to as traditional) toward end-to-end DL architectures. We provide a comprehensive list of the audiovisual databases available for lipreading, describing what tasks they can be used for, their popularity and their most important characteristics, such as the number of speakers, vocabulary size, recording settings and total duration. In correspondence with the shift toward DL, we show that there is a clear tendency toward large-scale datasets targeting realistic application settings and large numbers of samples per class. On the other hand, we summarize, discuss and compare the different ALR systems proposed in the last decade, separately considering traditional and DL approaches. We address a quantitative analysis of the different systems by organizing them in terms of the task that they target (e.g. recognition of letters or digits and words or sentences) and comparing their reported performance in the most commonly used datasets. As a result, we find that DL architectures perform similarly to traditional ones for simpler tasks but report significant improvements in more complex tasks, such as word or sentence recognition, with up to 40% improvement in word recognition rates. Hence, we provide a detailed description of the available ALR systems based on end-to-end DL architectures and identify a tendency to focus on the modeling of temporal context as the key to advance the field. Such modeling is dominated by recurrent neural networks due to their ability to retain context at multiple scales (e.g. short-and longterm information). In this sense, current efforts tend toward techniques that allow a more comprehensive modeling and interpretability of the retained context.

show abstract

Audio-Visual Speech Recognition Using Bimodal-Trained Bottleneck Features for a Person with Severe Hearing Loss

Cited by 22 publications

References 10 publications

Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models

Lip Reading Using Convolutional Neural Networks with and without Pre-Trained Models

Multi-channel Convolutional Neural Networks for Automatic Detection of Speech Deficits in Cochlear Implant Users

Survey on automatic lip-reading in the era of deep learning

Contact Info

Product

Resources

About