Monaural multi-talker speech recognition using factorial speech processing models

Abstract

A Pascal challenge entitled monaural multi-talker speech recognition was developed, targeting the problem of robust automatic speech recognition against speech like noises which significantly degrades the performance of automatic speech recognition systems. In this challenge, two competing speakers say a simple command simultaneously and the objective is to recognize speech of the target speaker. Surprisingly during the challenge, a team from IBM research, could achieve a performance better than human listeners on this task. The proposed method of the IBM team, consist of an intermediate speech separation and then a single-talker speech recognition. This paper reconsiders the task of this challenge based on gain adapted factorial speech processing models. It develops a joint-token passing algorithm for direct utterance decoding of both target and masker speakers, simultaneously. Comparing it to the challenge winner, it uses maximum uncertainty during the decoding which cannot be used in the past two-phased method. It provides detailed derivation of inference on these models based on general inference procedures of probabilistic graphical models. As another improvement, it uses deep neural networks for joint-speaker identification and gain estimation which makes these two steps easier than before producing competitive results for these steps. The proposed method of this work outperforms past superhuman results and even the results were achieved recently by Microsoft research, using deep neural networks. It achieved 5.5% absolute task performance improvement compared to the first superhuman system and 2.7% absolute task performance improvement compared to its recent competitor.

Feature joint-state posterior estimation in factorial speech processing models using deep neural networks

“…In the second scenario, two or more speaker voices are mixed together to produce a multi-talker speech utterance in which underlying source processes are the speakers' voices. In this case, previous achievements are surprising [6,7], even better than the results achieved manually by human listening (Fig. 8-left).…”

“…In fact, we assume that the initial solution to the system of equations with approximate joint-posteriors can be improved iteratively during the discriminative phase using marginal posteriors. Based on this assumption, we propose the following three steps for training a deep neural network for extracting joint-state posteriors: the generative phase, initializing joint-state layer weights, and 7 fine-tuning the network. Fig.…”

“…The joint-decoder objective is to find the most probable word sequences of the two speakers given features of the mixed-audio signal; we use all allowed information during the decoding in which the target speaker uses the word "white" in his command and the masker does not. The decoder solves the following optimization problem: Decoding of this task is done using a joint decoder implemented by the joint token passing algorithm proposed by [7]. The decoder reads HTK HParse generated wordnets generated based on the task grammar.…”

An Assessment of the Visual Features Extractions for the Audio-Visual Speech Recognition

“…VidTIMIT database covers 40 speaker's (22 guys and 18 females) as well as subset of this database having 30 speaker's (15 guys and 15 female's speaker's) was utilized in work depicted in this article. Every speaker expresses eight distinct sentences before a camera fixated on substance of speaker, and the sentences in database are on the whole instances of persistent discourse booked from the standard VidTIMIT database as well as comprise an aggregate of the 210 expressions and the terms of 920 words, and the sound is recorded at the test rate of 64 KHz and 32 bits profundity; video is recorded at the rate of 24 outlines for each second [10].…”

Reliability Modeling of Speech Recognition Tasks

“…Speech recognition technology has applications in different systems, such as automatic translation telephones, question and answer machines, and intelligence decisions support systems [1][2][3][4]. The mechanism of speech recognition lies in the separation of the words and the matching of patterns between the words in the speech and the words in a dictionary [5][6].…”