Deterministic annealing for trellis quantizer and HMM design using Baum-Welch re-estimation

Miller, David J.; Rose, Kenneth; Chou, Pang-Yen

doi:10.1109/icassp.1994.389482

Cited by 8 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The RD curve is obtained by minimizing the mutual information subject to an average distortion constraint. Formally stated, given a continuous source alphabet , random variable with a probability measure given by the density , and a reproduction alphabet , the problem is of that optimizing the mutual information (74) over the random encoders , subject to (75) where is the distortion measure. By replacing the above minimization with parametric variational equations (see [7], [12], [32], or [40]), this problem can be reformulated as a problem of optimization over the reproduction density .…”

Section: ) Mixture Of Expertsmentioning

confidence: 99%

See 1 more Smart Citation

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

1998

Self Cite

View full text Add to dashboard Cite

The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The method offers three important features: 1) the ability to avoid many poor local optima; 2) applicability to many different structures/architectures; and 3) the ability to minimize the right cost function even when its gradients vanish almost everywhere, as in the case of the empirical classification error. It is derived within a probabilistic framework from basic information theoretic principles (e.g., maximum entropy and random coding). The application-specific cost is minimized subject to a constraint on the randomness (Shannon entropy) of the solution, which is gradually lowered. We emphasize intuition gained from analogy to statistical physics, where this is an annealing process that avoids many shallow local minima of the specified cost and, at the limit of zero "temperature," produces a nonrandom (hard) solution. Alternatively, the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. This provides new insights into the method and its performance, as well as new insights into rate-distortion theory itself. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts. Experimental results show considerable performance gains over standard structure-specific and application-specific training methods. The paper concludes with a brief discussion of extensions of the method that are currently under investigation.

show abstract

Section: ) Mixture Of Expertsmentioning

confidence: 99%

“…One commonly used structure, particularly in speech coding, is the multistage vector quantizer (see [71] for an early DA approach). Another very important structure is the trellis quantizer (as well as the trellis vector quantizer) for which a DA approach has been proposed in [74]. …”

Section: B Extensions and Applicationsmentioning

confidence: 99%

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

1998

Self Cite

View full text Add to dashboard Cite

show abstract

“…As before, these conditions can be seen to specify a type of supervised learning rule which matches intuition. In (27) and (28) the in uence an individual classi cation decision has on the learning is a function of the associated risk L xj P jjx , as well as the sensitivity of the class output to the given parameter. In this case, the sensitivity of class output F j (x) to changes in a parameter from receptive eld k is ( kj (x) ?…”

Section: Optimality Conditions For Radial Basis Functionsmentioning

confidence: 99%

A global optimization technique for statistical classifier design

Miller

Rao

Rose

et al. 1996

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

A global optimization method is introduced for the design of statistical classi ers that minimize the rate of misclassi cation. We rst derive the theoretical basis for the method, based on which we develop a novel design algorithm and demonstrate its e ectiveness and superior performance in the design of practical classi ers for some of the most popular structures currently in use. The method, grounded in ideas from statistical physics and information theory, extends the deterministic annealing approach for optimization, both to incorporate structural constraints on data assignments to classes and to minimize the probability of error as the cost objective. During the design, data are assigned to classes in probability, so as to minimize the expected classi cation error given a speci ed level of randomness, as measured by Shannon's entropy. The constrained optimization is equivalent to a free energy minimization, motivating a deterministic annealing approach in which the entropy and expected misclassi cation cost are reduced with the temperature while enforcing the classi er's structure. In the limit, a hard classi er is obtained. This approach is applicable to a variety of classi er structures, including the widely used prototype-based, radial basis function, and multilayer perceptron classi ers. The method is compared with learning vector quantization, back propagation, several radial basis function design techniques, as well as with paradigms for more directly optimizing all these structures to minimize probability of error. The annealing method achieves signi cant performance gains over other design methods on a number of benchmark examples from the literature, while often retaining design complexity comparable to or only moderately greater than that of strict descent methods. Substantial gains, both inside and outside the training set, are achieved for complicated examples involving high-dimensional data and large class overlap.

show abstract

“…Recently, DA methods have been developed to design pattern classifiers [25], regression functions [36], [34] and a new class of source coding systems [35]. DA has previously been proposed for HMM design, albeit with ML as the design objective [26].…”

Section: Introductionmentioning

confidence: 99%

Deterministically annealed design of hidden Markov model speech recognizers

Rao

Rose

2001

IEEE Trans. Speech Audio Process.

Self Cite

View full text Add to dashboard Cite

Many conventional speech recognition systems are based on the use of hidden Markov models (HMM) within the context of discriminant-based pattern classification. While the speech recognition objective is a low rate of misclassification, HMM design has been traditionally approached via maximum likelihood (ML) modeling which is, in general, mismatched with the minimum error objective and hence suboptimal. Direct minimization of the error rate is difficult because of the complex nature of the cost surface, and has only been addressed recently by discriminative design methods such as generalized probabilistic descent (GPD). While existing discriminative methods offer significant benefits, they commonly rely on local optimization via gradient descent whose performance suffers from the prevalence of shallow local minima. As an alternative, we propose the deterministic annealing (DA) design method that directly minimizes the error rate while avoiding many poor local minima of the cost. DA is derived from fundamental principles of statistical physics and information theory. In DA, the HMM classifier's decision is randomized and its expected error rate is minimized subject to a constraint on the level of randomness which is measured by the Shannon entropy. The entropy constraint is gradually relaxed, leading in the limit of zero entropy to the design of regular nonrandom HMM classifiers. An efficient forward-backward algorithm is proposed for the DA method. Experiments on synthetic data and on a simplified recognizer for isolated English letters demonstrate that the DA design method can improve recognition error rates over both ML and GPD methods.Index Terms-Deterministic annealing, discriminative training, hidden Markov model, isolated word recognition, minimum classification error.

show abstract

Deterministic annealing for trellis quantizer and HMM design using Baum-Welch re-estimation

Cited by 8 publications

References 17 publications

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

A global optimization technique for statistical classifier design

Deterministically annealed design of hidden Markov model speech recognizers

Contact Info

Product

Resources

About