The accuracy of restricted Boltzmann machine models of Ising systems

Yevick, David; Melko, Roger G.

doi:10.1016/j.cpc.2020.107518

Cited by 14 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results are in agreement with the same quantities computed from samples of the Ising model distribution, see Figure 6B. This observation is consistent with literature [25,34,35], where RBMs were shown to be able to accurately fit statistical physics models such as the Ising model.…”

Section: B Learning With Standard Rbmsupporting

confidence: 91%

“…We trained RBMs with 200 -400 binary hidden units (see SI for details) on MNIST0/1, the 2-dimensional Ising model, and the KH domain protein family. Consistent with previous results on related datasets [18,19,25,26], RBM accuratetely fit and generate high-quality samples in the three cases (illustrated Figure 2C). In addition, training simple classifiers to predict the label from the hidden inputs of the models, gives an area under the curve (AUC) of > 0.9 in all cases; see Supporting information (SI Fig.…”

Section: Rbms Learn Distributed Representations Of the Labelsupporting

confidence: 90%

See 1 more Smart Citation

Disentangling representations in Restricted Boltzmann Machines without adversaries

Fernández-de-Cossio-Díaz¹,

Cocco²,

Monasson³

2022

Preprint

View full text Add to dashboard Cite

A goal of unsupervised machine learning is to disentangle representations of complex highdimensional data, allowing for interpreting the significant latent factors of variation in the data as well as for manipulating them to generate new data with desirable features. These methods often rely on an adversarial scheme, in which representations are tuned to avoid discriminators from being able to reconstruct specific data information (labels). We propose a simple, effective way of disentangling representations without any need to train adversarial discriminators, and apply our approach to Restricted Boltzmann Machines (RBM), one of the simplest representation-based generative models. Our approach relies on the introduction of adequate constraints on the weights during training, which allows us to concentrate information about labels on a small subset of latent variables. The effectiveness of the approach is illustrated on the MNIST dataset, the two-dimensional Ising model, and taxonomy of protein families. In addition, we show how our framework allows for computing the cost, in terms of log-likelihood of the data, associated to the disentanglement of their representations.

show abstract

Section: B Learning With Standard Rbmsupporting

confidence: 91%

Section: Rbms Learn Distributed Representations Of the Labelsupporting

confidence: 90%

Disentangling representations in Restricted Boltzmann Machines without adversaries

Fernández-de-Cossio-Díaz¹,

Cocco²,

Monasson³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, we want to mention that it is particularly surprising that the CD recipe (with short k) is still used [36,37,38,34]. As shown, RBMs trained with CD are not able to generate proper samples from scratch.…”

Section: Related Workmentioning

confidence: 99%

Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines

Decelle¹,

Furtlehner²,

Seoane³

2021

Preprint

View full text Add to dashboard Cite

Training Restricted Boltzmann Machines (RBMs) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient. Over the past decades, many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed to sample new configurations from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability of the trained model, and that RBMs operate in two well-defined regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of steps, k, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies a transition from one regime to another as soon as k becomes smaller than this time. In particular, we show that using the popular k (persistent) contrastive divergence approaches, with k small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display faster dynamics, and a smooth convergence to dataset-like configurations during the sampling. Finally we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short k can be used to generate convincing samples in short learning times, (ii) large k (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM.Preprint. Under review.

show abstract

“…This targeted output technique then displays a significant discrimination in latent space among digits even for two-dimensional latent spaces. Such behavior sets the procedure as well apart from other generative algorithms such as restricted Boltzmann machines [46] and principal component analysis, which generally require large latent space dimensions to achieve high accuracy. [47] The fixed target procedure could potentially be adapted to a variety of applications.…”

Section: Discussionmentioning

confidence: 99%

Rotated Digit Recognition by Variational Autoencoders with Fixed Output Distributions

Yevick

2022

Preprint

View full text Add to dashboard Cite

This paper demonstrates that a simple modification of the variational autoencoder (VAE) formalism enables the method to identify and classify rotated and distorted digits. In particular, the conventional objective (cost) function employed during the training process of a VAE both quantifies the agreement between the input and output data records and ensures that the latent space representation of the input data record is statistically generated with an appropriate mean and standard deviation. After training, simulated data realizations are generated by decoding appropriate latent space points. Since, however, standard VAE:s trained on randomly rotated MNIST digits cannot reliably distinguish between different digit classes since the rotated input data is effectively compared to a similarly rotated output data record. In contrast, an alternative implementation in which the objective function compares the output associated with each rotated digit to a corresponding fixed unreferenced reference digit is shown here to discriminate accurately among the rotated digits in latent space even when the dimension of the latent space is 2 or 3.

show abstract

The accuracy of restricted Boltzmann machine models of Ising systems

Cited by 14 publications

References 20 publications

Disentangling representations in Restricted Boltzmann Machines without adversaries

Disentangling representations in Restricted Boltzmann Machines without adversaries

Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines

Rotated Digit Recognition by Variational Autoencoders with Fixed Output Distributions

Contact Info

Product

Resources

About