Temporally Varying Weight Regression: A Semi-Parametric Trajectory Model for Automatic Speech Recognition

Abstract

Standard Hidden Markov Model (HMM) assumes that successive observations are independent to one another given the state sequence. This leads to a poor trajectory model for speech. Many explicit trajectory modeling techniques have been studied in the past to improve trajectory modeling for HMM. However, these techniques do not yield promising improvements over conventional HMM systems where differential parameters and Gaussian Mixture Model have been used implicitly to circumvent the poor trajectory modeling issue of HMM. Recently, semi-parametric trajectory modeling techniques based on temporally varying model parameters such as fMPE and pMPE have been shown to yield promising improvements over state-of-the-art systems on large vocabulary continuous speech recognition tasks. These techniques use high dimensional posterior features derived from a long span of acoustic features to model temporally varying attributes of the speech signal. Bases corresponding to these posterior features are then discriminatively estimated to yield temporally varying mean (fMPE) and precision matrix (pMPE) parameters. Motivated by the success of fMPE and pMPE, Temporally Varying Weight Regression (TVWR) was recently proposed to model HMM trajectory implicitly using time-varying Gaussian weights. In this paper, a complete formulation of TVWR is given based on a probabilistic modeling framework. Parameter estimation formulae using both maximum likelihood (ML) and minimum phone error (MPE) criteria are derived. Experimental results based on the Wall Street Journal (CSR-WSJ WSJ ) and Aurora 4 corpora show that consistent promising improvements over the standard HMM systems can be obtained in both the 20 k open vocabulary recognition task (NIST Nov'92 WSJ0) and 5 k closed vocabulary noisy speech recognition for both ML and MPE criteria.

Multi-frame factorisation for long-span acoustic modelling

“…The idea of factorizing the distribution of p(y t |q t , M) is not new. For instance, the semi-parametric trajectory model [19], rewrites the joint distribution as…”

“…, o t+k ), and we have used M τ , and M o to show that the two distributions may be modelled by different sets of model parameters. In [19], the model is simplified by dropping the dependency on o t as…”

“…Compared to standard HMMs (equation (7)), this approach predicts the states by averaging scores from a wider context. Note that the factorisation can be performed at the state level or the Gaussian component level [19]. This approach does not rely on a particular type of acoustic model, and a similar approach may be applied to DNNs [20], where the state posterior probability is obtained as…”

On combining DNN and GMM with unsupervised speaker adaptation for robust automatic speech recognition

“…Therefore, to get the best of both worlds, this paper proposes combining the GMMs and DNNs using the TVWR framework [13,14]. According to this framework, the state output probability of the long span acoustic features is given as:…”

“…In this paper, GMM+DNN/HMM is proposed as a novel system that combines the GMM and DNN using the Temporally Varying Weight Regression (TVWR) framework [13,14]. Based on this framework, a regression model is trained to transform the DNN posteriors into the time-varying scaling factors for the Gaussian weights.…”