The high-order Boltzmann machine: learned distribution and topology

Albizuri, F. X.; D’Anjou, Alicia; Graña, Manuel; Torrealdea, Javier; Hernandez, Maritza

doi:10.1109/72.377984

Cited by 19 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where D[q, p(ξ p )] is treated as a function of BM's parameters ξ p and λ is the learning rate. As shown in [32], the gradient descent method converges to the minimum of the divergence with proper choices of λ, and hence achieves the projection point Γ B (q). Last, we show that the mixed coordinates [ζ xh ] Γ B (q) in Equation ( 20) is exactly the convergence point of the ML learning for BM.…”

Section: Appendix H Proof Ofmentioning

confidence: 95%

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Zhao

Hou

Song

et al. 2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented dimensionality reduction in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called Confident-Information-First (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighbourhood of the underlying real distribution. Then we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann Machines (BM) from a model selection perspective and theoretically show that both the fully visible BM (VBM) and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the non-essential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.

show abstract

Section: Appendix H Proof Ofmentioning

confidence: 95%

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Zhao

Hou

Song

et al. 2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Let be a positive probability function on that admits a factorization (2) where 1 . It can be shown that the probability function can be written as through a consensus function where the weights are determined and the set of weighted connections is (3) We name the hypergraph of the factorization (2). In this notation is the set of indexes of the variables in .…”

Section: Structure Of the Hobm From Conditional Independencesmentioning

confidence: 99%

“…T HE conventional Boltzmann machine (BM) [1], [9], as well as the high-order Boltzmann machine (HOBM) [15], [3], is a technique whose purpose is, in its fundamental formulation, to describe and model probability distributions defined on a set of binary random variables.…”

Section: Introductionmentioning

confidence: 99%

Structure of the high-order Boltzmann machine from independence maps

Albizuri

D’Anjou

Graña

et al. 1997

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

In this paper we consider the determination of the structure of the high-order Boltzmann machine (HOBM), a stochastic recurrent network for approximating probability distributions. We obtain the structure of the HOBM, the hypergraph of connections, from conditional independences of the probability distribution to model. We assume that an expert provides these conditional independences and from them we build independence maps, Markov and Bayesian networks, which represent conditional independences through undirected graphs and directed acyclic graphs respectively. From these independence maps we construct the HOBM hypergraph. The central aim of this paper is to obtain a minimal hypergraph. Given that different orderings of the variables provide in general different Bayesian networks, we define their intersection hypergraph. We prove that the intersection hypergraph of all the Bayesian networks (N!) of the distribution is contained by the hypergraph of the Markov network, it is more simple, and we give a procedure to determine a subset of the Bayesian networks that verifies this property. We also prove that the Markov network graph establishes a minimum connectivity for the hypergraphs from Bayesian networks.

show abstract

“…Topologies are widely used in the research field of machine learning and cybernetics (see e.g. Acencio and Lemke 2009;Albizuri et al 1995;Carr et al 2009;Choudhury and Zaman 2006;Kall et al 2004;Koretelainen 1999Koretelainen , 2001Li et al 2003;Thierens and Vercauteren 1991). For example, Koretelainen (1999Koretelainen ( , 2001 used topologies to detect dependencies of attributes in information systems with respect to gradual rules.…”

Section: Introductionmentioning

confidence: 99%

On intuitionistic fuzzy topologies based on intuitionistic fuzzy reflexive and transitive relations

Zhou

2010

Soft Comput

View full text Add to dashboard Cite

Topologies and rough set theory are widely used in the research field of machine learning and cybernetics. An intuitionistic fuzzy rough set, which is the result of approximation of an intuitionistic fuzzy set with respect to an intuitionistic fuzzy approximation space, is an extension of fuzzy rough sets. For further studying the theories and applications of intuitionistic fuzzy rough sets, in this paper, we investigate the topological structures of intuitionistic fuzzy rough sets. We show that an intuitionistic fuzzy rough approximation space can induce an intuitionistic fuzzy topological space in the sense of Lowen if and only if the intuitionistic fuzzy relation in the approximation space is reflexive and transitive. We also examine the sufficient and necessary conditions that an intuitionistic fuzzy topological space can be associated with an intuitionistic fuzzy reflexive and transitive relation such that the induced lower and upper intuitionistic fuzzy rough approximation operators are, respectively, the intuitionistic fuzzy interior and closure operators of the given topology.

show abstract

The high-order Boltzmann machine: learned distribution and topology

Cited by 19 publications

References 10 publications

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

A Confident Information First Principle for Parameter Reduction and Model Selection of Boltzmann Machines

Structure of the high-order Boltzmann machine from independence maps

On intuitionistic fuzzy topologies based on intuitionistic fuzzy reflexive and transitive relations

Contact Info

Product

Resources

About