A global optimization technique for statistical classifier design

Abstract

A global optimization method is introduced for the design of statistical classi ers that minimize the rate of misclassi cation. We rst derive the theoretical basis for the method, based on which we develop a novel design algorithm and demonstrate its e ectiveness and superior performance in the design of practical classi ers for some of the most popular structures currently in use. The method, grounded in ideas from statistical physics and information theory, extends the deterministic annealing approach for optimization, both to incorporate structural constraints on data assignments to classes and to minimize the probability of error as the cost objective. During the design, data are assigned to classes in probability, so as to minimize the expected classi cation error given a speci ed level of randomness, as measured by Shannon's entropy. The constrained optimization is equivalent to a free energy minimization, motivating a deterministic annealing approach in which the entropy and expected misclassi cation cost are reduced with the temperature while enforcing the classi er's structure. In the limit, a hard classi er is obtained. This approach is applicable to a variety of classi er structures, including the widely used prototype-based, radial basis function, and multilayer perceptron classi ers. The method is compared with learning vector quantization, back propagation, several radial basis function design techniques, as well as with paradigms for more directly optimizing all these structures to minimize probability of error. The annealing method achieves signi cant performance gains over other design methods on a number of benchmark examples from the literature, while often retaining design complexity comparable to or only moderately greater than that of strict descent methods. Substantial gains, both inside and outside the training set, are achieved for complicated examples involving high-dimensional data and large class overlap.

Use of Identification Trial Statistics for the Combination of Biometric Matchers

“…Though previous research proposing solutions for this problem exists (e.g., [31], [32]), it deals with the case of small number of classes and is not directly applicable to our case. The key idea of learning algorithms minimizing classification error is to replace the discrete misclassification cost function with some smooth approximation in order to be able to take a derivative of the cost function and perform gradient descent optimization.…”

Error-resilient and complexity-constrained distributed coding for large scale sensor networks

“…In general, higher correlations assist the bit-mapper as it uses all the received bits to correct errors, unlike the grouping approach which is forced to use only the bits within each group. From 4(b), it also follows that the overhead required to store the Bayesian network aggravates at higher N and the performance degrades further, making the Bayesian network approach impractical for very large networks 7 . Also, for these datasets, observe that the performance of the greedy-iterative descent method is considerably poorer than that using DA.…”

“…Further, we impose the 'nearest prototype' structural constraint on the bit-mapper partitions by appropriately choosing a parametrization of the association probabilities. Similar methods have been used before in the context of design of tree-structured quantizers [13], generalized VQ design [11] and optimal classifier design [7]). It can be shown using the principle of entropy maximization that (refer to [13]), to impose a 'nearest prototype' structure, at each temperature, the association probabilities must be governed by the Gibbs distribution:…”

“…As the total storage is not reflected in these plots, we do not consider the performance of the Bayesian network approach hereafter, noting that, the storage required to achieve good distortion performance is significantly higher. 7 For the rainfall dataset, the storage required for the Bayesian network approach was significantly larger and hence we do not plot it along with the other curves…”

Annealed Discriminant Analysis

“…The deterministic annealing (DA) technique has demonstrated substantial performance improvement over clustering, classification and constrained optimization problems [1,2,3,4,5]. Since DA is strongly motivated by the analogies to statistical physics [6], it regards the optimization problem in question as a thermal system.…”