The α-EM algorithm: surrogate likelihood maximization using α-logarithmic information measures

Abstract. Measures of divergence between two points play a key role in many engineering problems. One such measure is a distance function, but there are many important measures which do not satisfy the properties of the distance. The Bregman divergence, KullbackLeibler divergence and f -divergence are such measures. In the present article, we study the differential-geometrical structure of a manifold induced by a divergence function. It consists of a Riemannian metric, and a pair of dually coupled affine connections, which are studied in information geometry. The class of Bregman divergences are characterized by a dually flat structure, which is originated from the Legendre duality. A dually flat space admits a generalized Pythagorean theorem. The class of f -divergences, defined on a manifold of probability distributions, is characterized by information monotonicity, and the Kullback-Leibler divergence belongs to the intersection of both classes. The f -divergence always gives the α-geometry, which consists of the Fisher information metric and a dual pair of ±α-connections. The α-divergence is a special class of f -divergences. This is unique, sitting at the intersection of the f -divergence and Bregman divergence classes in a manifold of positive measures. The geometry derived from the Tsallis q-entropy and related divergences are also addressed.

show abstract

“…Its applications were described earlier by Chernoff in 1952 [33], and later in [34,35] etc. to mention a few.…”

Section: Information Geometry Of Divergence Functionsmentioning

confidence: 91%

Information geometry of divergence functions

Амари¹,

Cichocki²

2010

Bulletin of the Polish Academy of Sciences: Technical Sciences

107

152

View full text Add to dashboard Cite

show abstract

“…Besides the RapidICA of this paper, further variants are possible, some of which might show performances comparable to that of the RapidICA. We also note here that the surrogate optimization used in the alpha-HMM [12] originates in the same place [10] as does the RapidICA. …”

Section: Total Algorithm: the Rapidicamentioning

confidence: 99%

“…The use of a momentum term appears in various iterative methods. The FastICA is not JSIP an exception, since acceleration methods for the natural gradient ICA have already been presented [3][4][5] based upon the idea of surrogate optimization of the likelihood ratio [10]. In fact, the RapidICA presented in this paper is an embodiment of the momentum term of the α-ICA applied to the fixed-point ICA expressed by the additive form (19).…”

Section: Total Algorithm: the Rapidicamentioning

confidence: 99%

See 1 more Smart Citation

Rapid Algorithm for Independent Component Analysis

Yokote¹,

Matsuyama²

2012

JSIP

View full text Add to dashboard Cite

A class of rapid algorithms for independent component analysis (ICA) is presented. This method utilizes multi-step past information with respect to an existing fixed-point style for increasing the non-Gaussianity. This can be viewed as the addition of a variable-size momentum term. The use of past information comes from the idea of surrogate optimization. There is little additional cost for either software design or runtime execution when past information is included. The speed of the algorithm is evaluated on both simulated and real-world data. The real-world data includes color images and electroencephalograms (EEGs), which are an important source of data on human-computer interactions. From these experiments, it is found that the method we present here, the RapidICA, performs quickly, especially for the demixing of super-Gaussian signals.

show abstract

“…Notice that the soft clustering approach learns all parameters, including λ (if not constrained to zero or one) and α ∈ R. This is not the case for Matsuyama's α-expectation maximization (EM) algorithm [42] in which α is fixed beforehand (and, thus, not learned).…”

Section: Soft Mixed α-Clusteringmentioning

confidence: 99%

On Clustering Histograms with k-Means by Using Mixed α-Divergences

Nielsen

Nock

Амари

2014

Entropy

View full text Add to dashboard Cite

Clustering sets of histograms has become popular thanks to the success of the generic method of bag-of-X used in text categorization and in visual categorization applications. In this paper, we investigate the use of a parametric family of distortion measures, called the α-divergences, for clustering histograms. Since it usually makes sense to deal with symmetric divergences in information retrieval systems, we symmetrize the α-divergences using the concept of mixed divergences. First, we present a novel extension of k-means clustering to mixed divergences. Second, we extend the k-means++ seeding to mixed α-divergences and report a guaranteed probabilistic bound. Finally, we describe a soft clustering technique for mixed α-divergences.

show abstract

The α-EM algorithm: surrogate likelihood maximization using α-logarithmic information measures

Cited by 41 publications

References 22 publications

Information geometry of divergence functions

Information geometry of divergence functions

Rapid Algorithm for Independent Component Analysis

On Clustering Histograms with k-Means by Using Mixed α-Divergences

Contact Info

Product

Resources

About