Classification of Binary Vectors by Stochastic Complexity

Abstract

Stochastic complexity is treated as a tool of classification, i.e., of inferring the number of classes, the class descriptions, and the class memberships for a given data set of binary vectors. The stochastic complexity is evaluated with respect to the family of statistical models defined by finite mixtures of multivariate Bernoulli distributions obtained by the principle of maximum entropy. It is shown that stochastic complexity is asymptotically related to the classification maximum likelihood estimate. The formulae for stochastic complexity have an interpretation as minimum code lengths for certain universal source codes for storing the binary data vectors and their assignments into the classes in a classification. There is also a decomposition of the classification uncertainty in a sum of an intraclass uncertainty, an interclass uncertainty, and a special parsimony term. It is shown that minimizing the stochastic complexity amounts to maximizing the information content of the classification. An algorithm of alternating minimization of stochastic complexity is given. We discuss the relation of the method to the AUTOCLASS system of Bayesian classification. The application of classification by stochastic complexity to an extensive data base of strains of Enterobacteriaceae is described.

On combining multiple clusterings: an overview and a new perspective

“…Binary data clustering has been widely studied in literature [25,29,33,42]. A unified view of binary data clustering has been provided by examining the connections among various methods including entropy-based methods, distance-based methods (e.g., K-means), mixture models, and matrix decomposition [38,39].…”

Classification of Enterobacteriaceae by minimization of stochastic complexity

“…The first term in equation (3) describes the complexity of the classification and the second term the complexity of the strains with respect to the classification. Gyllenberg et al (1994b) also showed that minimizing the SC with respect to the model (2) amounts to maximizing the information content of the classification.…”

“…Gyllenberg et al (1994b) showed that minimizing SC amounts to maximizing the information content of the classification. Thus increasing SC implies loss of information whereas decreasing SC indicates gain in information content.…”

“…It was shown by Gyllenberg et al (1994b) that the stochastic complexity SC of a set of t strains with respect to the above model is where ti is the number of strains in class j and tij is the number of strains in class j with the it" feature equal to 1 (log denotes the logarithm to the base 2). The first term in equation (3) describes the complexity of the classification and the second term the complexity of the strains with respect to the classification.…”

“…Gyllenberg et al (1994b) The procedure is repeated starting from step 2 using the HMOs found in step 4 at the previous iteration until the HMOs do not change.…”