Online Deterministic Annealing for Classification and Clustering

Abstract

We introduce an online prototype-based learning algorithm for clustering and classification, based on the principles of deterministic annealing. We show that the proposed algorithm constitutes a competitive-learning neural network, the learning rule of which is formulated as an online stochastic approximation algorithm. The annealing nature of the algorithm prevents poor local minima, offers robustness with respect to the initial conditions, and provides a means to progressively increase the complexity of the learning model as needed, through an intuitive bifurcation phenomenon. As a result, the proposed approach is interpretable, requires minimal hyper-parameter tuning, and offers online control over the complexity-accuracy trade-off. Finally, Bregman divergences are used as a family of dissimilarity measures that are shown to play an important role in both the performance of the algorithm, and its computational complexity. We illustrate the properties and evaluate the performance of the proposed learning algorithm in artificial and real datasets.

Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach

“…To define an online training rule for the deterministic annealing framework, we formulate a stochastic approximation algorithm to recursively estimate E [X|µ] directly. As a direct consequence of Theorem 4 in [30], the following corollary provides an online learning rule that solves the optimization problem of the deterministic annealing algorithm.…”

“…However, their convergence properties and final configuration depend heavily on two design parameters: (a) the number of clusters (neurons), and (b) their initial configuration. To deal with this phenomenon, the Online Deterministic Annealing approach [30] makes use of a probabilistic framework, where input vectors are assigned to clusters in probability, thus dropping the assumption that Q is a deterministic function of X. For the randomized partition, the expected distortion becomes:…”

“…the Lagrange multiplier of the multi-objective minimization problem in (12). As shown in [30], we can detect the bifurcation points by maintaining and perturbing pairs of codevectors at each effective cluster so that they separate only when a critical temperature is reached.…”

“…In particular, the family of Bregman divergences, which includes the widely used Kullback-Leibler divergence, can offer numerous advantages in learning applications compared to the Euclidean distance alone [34]. Notably, in the case of deterministic annealing, Bregman divergences play an even more important role, since we can show that, if d is a Bregman divergence, the solution to the second optimization step ( 14) can be analytically computed in a convenient centroid form [30]:…”

“…Regarding (iii), we build upon the Online Deterministic Annealing (ODA) algorithm [30], a novel learning algorithm that constructs progressively growing knowledge representations. ODA can be formulated as a series of soft-clustering optimization problems…”