A hybrid segmental neural net/hidden Markov model system for continuous speech recognition

Abstract

Absfrucf-The current state-of-the-art in large-vocabulary, continuous speech recognition is based on the use of hidden Markov models (HMM). In an attempt to improve over HMM performance, we developed a hybrid system that combines the advantages of neural networks and HMM using a multiple hypothesis (or N-best) paradigm. The connectionist component of the system, the segmental neural net (SNN), models all the frames of a phonetic segment simultaneously, thus overcoming the well-known conditional-independence limitation of HMM's. In the paper, we describe the hybrid system and we discuss various aspects of SNN modeling, including network architectures, training algorithms and context modeling. Finally, we evaluate the hybrid system by performing several speaker-independent experiments with the DARPA Resource Management (RM) corpus, and we demonstrate that the hybrid system shows a consistent improvement in performance over the baseline HMM system. I. I NTRODUCTI ONONTINUOUS speech recognition (CSR) is in principle C a massive search procedure among all possible sentences (word sequences) allowed by the vocabulary and the grammar, and all possible alignments of each sentence with the input speech, to find the sentence and the alignment that are the most likely given the input speech. Because of the variability of individual words in time and the fact that words are connected, a successful CSR system must deal with many interrelated aspects of time modeling: the variability of parameters within a speech unit (such as a word or a phoneme) as a function of time, the variability that is introduced by the acoustic dependence of one phoneme on neighboring phonemes, and the need to model the input speech as a sequence of concatenated speech units, allowing for different possible alignments or segmentations. Finally, for each possible alignment between a hypothesized sequence of speech units and the input speech, the system must produce a global match score to indicate how closely it matches the input speech. The global score consists of a combination of the scores of the individual speech units.State-of-the-art CSR systems are based on the use of hidden Markov models (HMM) to model phonemes in context. Fig. 1 gives a summary of the basic functioning of a HMM and how it deals with the time modeling problems stated above.' Basic to are with BBN Systems and IEEE Log Number 9214815.'An extensive description of HMM is out of the scope of this paper.The interest reader may refer to [l] for a tutorial, and to [2] for a more detailed presentation. An easily readable overview of an HMM-based speech recognition system can be found at [3].Technologies, Cambridge, MA 02138. spectral frat" Fig. 1. Basic functioning of a hidden Markov model (HMM). Model: Thephonetic models use a number of states (three in this example) and left to right transitions. Associated with each transition is the probability of moving from one state to another. Associated with each state is a probability density function (pdf) the spectral features p(xlstate), f...

A survey of hybrid ANN/HMM models for automatic speech recognition

“…The idea, proposed in [130], is developed in [132], where a connectionist approach is applied to the problem of rescoring the hypothesis generated by an HMM which uses an N-best strategy. In this case the network does not compute scores on individual acoustic frames, but on whole segments (sub-sequencies) of frames, corresponding to phonemes.…”

“…Di!erent neural architectures were tried (1-layer perceptron, MLP, HyperBF) [132] without signi"cant #uctuations in performance. They were also combined altogether to obtain a more robust rescoring process.…”

“…Connectionist rescoring [130,132] A segmental neural network computes scores on whole segments of frames, corresponding to phonemes, for re-scoring the segmentation hypothesis yielded by an HMM which uses an N-best strategy. Rescoring of confusable wordsa [78] MLPs are applied after the HMM has generated the N-best word-sequence hypothesis, to`correcta (or con"rm) those individual words that belong to speci"c`confusablea classes.…”

A Discriminative Segmental Speech Model and Its Application to Hungarian Number Recognition

“…The most popular solution for avoiding the problems associated with P (S|A) is to run a frame-based (e.g. HMM) recognizer, and re-score only the N best paths by the segmental phoneme models [11]. We, however, wanted to model P (S|A) with discriminative classifiers, for which we trained segmental probabilities P (s i |A i ).…”

Posterior-based confidence measures for spoken term detection

“…MLP-based posteriors have also been used to re-score hypothesis in continuous speech recognition [4].…”