Information geometry of Boltzmann machines

Амари, Шун-ичи; Kurata, K.; Nagaoka, Hiroshi

doi:10.1109/72.125867

Cited by 171 publications

(158 citation statements)

References 20 publications

Supporting

Mentioning

156

Contrasting

Order By: Relevance

“…In particular, some authors [6,51,54] reported applications of information geometry to Bayesian statistics [32] and Bartlett correction [15]. In addition, we must say that information geometry is applied not just to statistics and information theory but also to combinatorics of neural networks and psychology [5], physics [20] and many other fields.…”

Section: Information Geometrymentioning

confidence: 99%

A five-dimensional Riemannian manifold with an irreducible SO(3)-structure as a model of abstract statistical manifold

Mikeš

Stepanova

2013

Ann Glob Anal Geom

View full text Add to dashboard Cite

In the present paper, we consider a five-dimensional Riemannian manifold with an irreducible SO(3)-structure as an example of an abstract statistical manifold. We prove that if a five-dimensional Riemannian manifold with an irreducible SO(3)-structure is a statistical manifold of constant curvature, then the metric of the Riemannian manifold is an Einstein metric. In addition, we show that a five-dimensional Euclidean sphere with an irreducible SO(3)-structure cannot be a conjugate symmetric statistical manifold. Finally, we show some results for a five-dimensional Riemannian manifold with a nearly integrable SO(3)-structure. For example, we prove that the structure tensor of a nearly integrable SO(3)-structure on a five-dimensional Riemannian manifold is a harmonic symmetric tensor and it defines the first integral of third order of the equations of geodesics. Moreover, we consider some topological properties of five-dimensional compact and conformally flat Riemannian manifolds with irreducible SO(3)-structure.

show abstract

Section: Information Geometrymentioning

confidence: 99%

A five-dimensional Riemannian manifold with an irreducible SO(3)-structure as a model of abstract statistical manifold

Mikeš

Stepanova

2013

Ann Glob Anal Geom

View full text Add to dashboard Cite

show abstract

“…As shown in [32], G θ and G η are mutually inverse matrices, i.e., J g IJ g JK = δ I K , where δ I K = 1 if I = K and zero if I = K. Next, we will develop the two propositions to compute G θ and G η generally. Note that Proposition 2.1 is a generalization of Theorem 2 in [33].…”

Section: B Fisher Information Matrix For Parametric Coordinatesmentioning

confidence: 94%

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Zhao

Hou

Song

et al. 2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented dimensionality reduction in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called Confident-Information-First (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighbourhood of the underlying real distribution. Then we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann Machines (BM) from a model selection perspective and theoretically show that both the fully visible BM (VBM) and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the non-essential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.

show abstract

“…In the present study, we investigate the information geometry of RoBMs along the line of research by Amari [3]. First, we reveal that RoBMs form an exponential family and determine the Fisher metric, natural and expectation coordinate systems, and the potential functions of RoBMs.…”

Section: Introductionmentioning

confidence: 92%

“…Thus RoBMs realize more flexible distributions than CBMs. Amari has investigated Boltzmann machines through information geometry [2,3]. Information geometry is an excellent method for analyzing stochastic models [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Information geometry of rotor Boltzmann machines

Kobayashi

2016

NOLTA

View full text Add to dashboard Cite

Abstract:A complex-valued Hopfield neural network is a useful model for processing multilevel data. A rotor Hopfield network is an extension of a complex-valued Hopfield neural network but much more flexible. In addition, a rotor Hopfield neural network has excellent storage capacity and noise robustness characteristics. In the present work, we investigate the rotor Boltzmann machine (RoBM), a stochastic model of a rotor Hopfield neural network, through information geometry, which is a useful tool for analyzing stochastic models. We discuss RoBM through concepts of information geometry, such as the Fisher metric, parameters and potential functions. Moreover, we provide natural gradient descent learning and emalgorithms for RoBM as applications of information geometry.

show abstract

Information geometry of Boltzmann machines

Cited by 171 publications

References 20 publications

A five-dimensional Riemannian manifold with an irreducible SO(3)-structure as a model of abstract statistical manifold

A five-dimensional Riemannian manifold with an irreducible SO(3)-structure as a model of abstract statistical manifold

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Information geometry of rotor Boltzmann machines

Contact Info

Product

Resources

About