Temperature based Restricted Boltzmann Machines

Li, Guoqi; Deng, Lei; Xu, Yi; Chen, Wen; Wang, Wei; Pei, Jing; Shi, Luping

doi:10.1038/srep19133

Cited by 28 publications

(23 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One can observe the best results were obtained by DBM when using T ∈ {0.1, 0.2, 0.5}. Also, DBN-CD benefit from lower temperatures, thus confirming the results obtained by Lin et al [16], i.e. the lower the temperature the higher the entropy.…”

Section: Resultssupporting

confidence: 86%

See 1 more Smart Citation

Temperature-Based Deep Boltzmann Machines

Passos

Papa

2017

Neural Process Lett

View full text Add to dashboard Cite

Deep learning techniques have been paramount in the last years, mainly due to their outstanding results in a number of applications, that range from speech recognition to face-based user identification. Despite other techniques employed for such purposes, Deep Boltzmann Machines (DBMs) are among the most used ones, which are composed of layers of Restricted Boltzmann Machines stacked on top of each other. In this work, we evaluate the concept of temperature in DBMs, which play a key role in Boltzmann-related distributions, but it has never been considered in this context up to date. Therefore, the main contribution of this paper is to take into account this information, as well as the impact of replacing a standard Sigmoid function by another one and to evaluate their influence in DBMs considering the task of binary image reconstruction. We expect this work can foster future research considering the usage of different temperatures during learning in DBMs.

show abstract

Section: Resultssupporting

confidence: 86%

“…https://archive.ics.uci.edu/ml/datasets/Semeion+Handwritten+Digit 7. Similar architectures have been commonly employed in the literature[12,16,24,25,34] 8. Notice all parameters and architectures have been empirically chosen[21].…”

mentioning

confidence: 99%

Temperature-Based Deep Boltzmann Machines

Passos

Papa

2017

Neural Process Lett

View full text Add to dashboard Cite

show abstract

“…Its impact is evaluated through the learning steps, and the results are compared even with distinct activation functions, once such parameter added to the energy function can be interpreted as a scalar multiplication of the Sigmoid function input. Provided results confirm the hypothesis suggested by Li et al [13] that lower temperatures tend to reach more accurate results, as presented in Table I. Furthermore, one can observe that lower temperatures also support sparseness representations of the hidden layer, which leads to a dropout like regularization.…”

Section: Temperature-based Deep Boltzmann Machinessupporting

confidence: 89%

On the Training Algorithms for Restricted Boltzmann Machines

Passos

Papa

2019

Anais Estendidos Do XXXII Conference on Graphics, Patterns and Images (SIBRAPI Estendido 2019)

View full text Add to dashboard Cite

Deep learning techniques have been studied extensively in the last years due to their good results related to essential tasks on a large range of applications, such as speech and face recognition, as well as object classification. Restrict Boltzmann Machines (RBMs) are among the most employed techniques, which are energy-based stochastic neural networks composed of two layers of neurons whose objective is to estimate the connection weights between them. Recently, the scientific community spent much effort on sampling methods since the effectiveness of RBMs is directly related to the success of such a process. Thereby, this work contributes to studies concerning different training algorithms for RBMs, as well as its variants Deep Belief Networks and Deep Boltzmann Machines. Further, the work covers the application of meta-heuristic methods concerning a proper fine-tune of these techniques. Moreover, the validation of the model is presented in the context of image reconstruction and unsupervised feature learning. In general, we present different approaches to training these techniques, as well as the evaluation of meta-heuristic methods for fine-tuning parameters, and its main contributions are: (i) temperature parameter introduction in DBM formulation, (ii) DBM using adaptive temperature, (iii) DBM meta-parameter optimization through meta-heuristic techniques, and (iv) infinity Restricted Boltzmann Machine (iRBM) meta-parameters optimization through meta-heuristic techniques.

show abstract

“…The even or odd sector of momentum values correspond to whether periodic or antiperiodic boundary conditions are imposed on the free fermion operator. We interpret the input data, vectors |i of 256 grayscale values for each pixel, as the eigenstates of the Hamiltonian (21). In order to do that, we binarize the MNIST data by setting a pixel value to 0 if it is smaller than 256/2, and 1 otherwise.…”

Section: A Simple Gge Machine For the Mnist Datasetmentioning

confidence: 99%

“…While there is a computational cost associated to calculating the charges we feed to the network, this is still a decrease in the total cost. The key difference here is the fact that the GGE algorithm assumes a simple Hamiltonian (21) with homogeneous coupling, whereas the RBM learns an inhomogeneous Hamiltonian with many different coupling constants.…”

Section: The Algorithm and Performance On The Mnist Datasetmentioning

confidence: 99%

Machine learning algorithms based on generalized Gibbs ensembles

Puskarov¹,

Cubero²

2018

J. Stat. Mech.

View full text Add to dashboard Cite

Machine learning algorithms often take inspiration from the established results and knowledge from statistical physics. A prototypical example is the Boltzmann machine algorithm for supervised learning, which utilizes knowledge of classical thermal partition functions and the Boltzmann distribution. Recently, a quantum version of the Boltzmann machine was introduced by Amin, et. al., however, noncommutativity of quantum operators renders the training process by minimizing a cost function inefficient. Recent advances in the study of non-equilibrium quantum integrable systems, which never thermalize, have lead to the exploration of a wider class of statistical ensembles. These systems may be described by the so-called generalized Gibbs ensemble (GGE), which incorporates a number of "effective temperatures". We propose that these GGEs can be successfully applied as the basis of a Boltzmann-machine-like learning algorithm, which operates by learning the optimal values of effective temperatures. We show that the GGE algorithm is an optimal quantum Boltzmann machine: it is the only quantum machine that circumvents the quantum training-process problem. We apply a simplified version of the GGE algorithm, where quantum effects are suppressed, to the classification of handwritten digits in the MNIST database. While lower error rates can be found with other state-of-the-art algorithms, we find that our algorithm reaches relatively low error rates while learning a much smaller number of parameters than would be needed in a traditional Boltzmann machine, thereby reducing computational cost. * t.puskarov@uu.nl † a.cortescubero@uu.nl arXiv:1804.03546v3 [cond-mat.stat-mech] 1 Dec 2018

show abstract

Temperature based Restricted Boltzmann Machines

Cited by 28 publications

References 60 publications

Temperature-Based Deep Boltzmann Machines

Temperature-Based Deep Boltzmann Machines

On the Training Algorithms for Restricted Boltzmann Machines

Machine learning algorithms based on generalized Gibbs ensembles

Contact Info

Product

Resources

About