A source coding approach to classification by vector quantization and the principle of minimum description length

Abstract

An algorithm for supervised classi cation using vector quantization and entropy coding is presented. The classi cation rule is formed from a set of training data fX i ; Y i g n i=1 , which are independent samples from a joint distribution P X Y . Based on the principle of Minimum Description Length MDL, a statistical model that approximates the distribution P X Y ought to enable ecient coding of X and Y . On the other hand, we expect a system that encodes X;Y e ciently to provide ample information on the distribution P X Y . This information can then be used to classify X , i.e., to predict the corresponding Y based on X . To encode both X and Y , a t wo-stage vector quantizer is applied to X and a Hu man code is formed for Y conditioned on each quantized value of X . The optimization of the encoder is equivalent to the design of a vector quantizer with an objective function re ecting the joint penalty of quantization error and misclassi cation rate. This vector quantizer provides an estimation of the conditional distribution of Y given X , which in turn yields an approximation to the Bayes classi cation rule. This algorithm, namely Discriminant Vector Quantization DVQ, is compared with Learning Vector Quantization LVQ and CART R on a number of data sets. DVQ outperforms the other two on several data sets. The relation between DVQ, density estimation, and regression is also discussed. I IntroductionThe goal of supervised classi cation is to form decision rules based on a set of training data so that misclassi cation penalty is minimized. Assume the training set is L = fx i ; y i ; i = 1; 2; :::; ng, where x i ; y i 's are independent samples from a joint distribution P X Y , X 2 X, Y 2 Y. Typically X is a continuous random variable

Classification via Minimum Incremental Coding Length

“…For instance, when detecting a face in an image, features associated with the face often have a low-dimensional structure which is "embedded" as a submanifold in a cloud of essentially random features from the background. Model selection criteria such as the minimum description length (MDL) [28,22] serve as important modifications to MAP for estimating a model across classes of different complexity. MDL selects the model that minimizes the overall coding length of the given (training) data, hence the name "minimum description length" or "minimum coding length" [1].…”

A Kernel Density Estimation Based Interestingness Measure for Association Rule Mining

“…However, it is hard to conclude they are with the same interestingness. Intuitively, c→ C 0 is the most interesting one, for it only contains 1 item in this antecedent and is favored by the Minimal Description Length Principle [8]. The situation for other four rules is more complicated and a systemic measure is deserved.…”