An Infinite Restricted Boltzmann Machine

Côté, Marc-Alexandre; Larochelle, Hugo

doi:10.1162/neco_a_00848

Cited by 52 publications

(63 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can also be further approximated with the function ln(1 + e s ). This method has been used in the past to develop models such as the Infinite RBM and Rate-coded RBM in generative machine learning [53,54].…”

Section: Generative Bi-partite Modelmentioning

confidence: 99%

A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data

Wong

Farooq

2020

Transportation Research Part C: Emerging Technologies

View full text Add to dashboard Cite

The emergence of data-driven demand analysis have led to the increased use of generative modelling to learn the probabilistic dependencies between random variables. Although their apparent use has largely been limited to image recognition and classification in recent years, generative machine learning algorithms can be a powerful tool for travel behaviour research by replicating travel behaviour by the underlying properties of data structures. In this paper, we examine the use of generative machine learning approach for analyzing multiple discrete-continuous (MDC) travel behaviour data. We provide a plausible perspective of how we can exploit the use of machine learning techniques to interpret the underlying heterogeneities in the data. We show that generative models are conceptually similar to choice selection behaviour process through information entropy and variational Bayesian inference. Without loss of generality, we consider a restricted Boltzmann machine (RBM) based algorithm with multiple discrete-continuous layer, formulated as a variational Bayesian inference optimization problem. We systematically describe the proposed machine learning algorithm and develop a process of analyzing travel behaviour data from a generative learning perspective. We show parameter stability from model analysis and simulation tests on an open dataset with multiple discrete-continuous dimensions from a data size of 293,330 observations. For interpretability, we derive the conditional probabilities, elasticities and perform statistical analysis on the latent variables. We show that our model can generate statistically similar data distributions for travel forecasting and prediction and performs better than purely discriminative methods in validation. Our results indicate that latent constructs in generative models can accurately represent the joint distribution consistently on MDC data.

show abstract

Section: Generative Bi-partite Modelmentioning

confidence: 99%

A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data

Wong

Farooq

2020

Transportation Research Part C: Emerging Technologies

View full text Add to dashboard Cite

show abstract

“…Max-norm regularization [16] was also used to suppress very large weights, the bounds for each i W  and i U  were 10 and 5 respectively. Côté and Larochelle [14] claims that results of learning are robust to the value of the hidden unit penalty i  . We have tried several different i  and find that smaller i  enables the model to grow to proper size faster at the beginning of learning.…”

Section: Evaluation Of the Modelsmentioning

confidence: 99%

“…Nair，et al [13] conceptually tie the weights of an infinite number of binary hidden units, and connect these sigmoid units with noisy rectified linear units (ReLUs) for better feature learning. More recently, Côté and Larochelle [14] have proposed a non-parametric model called the iRBM. By making the effective number of hidden units participating in the energy function change freely during training, the iRBM can automatically adjust the effective number of hidden units according to the data.…”

Section: Introductionmentioning

confidence: 99%

“…The reason for this drawback is that, the hidden units are correlated with each other given the visible variables. The learned filters or feature detectors change slowly from the left-most hidden unit to right, which is also called "ordering effect" in [14].…”

Section: Introductionmentioning

confidence: 99%

“…The hidden units of iRBM are selected in sequence as z takes the value from 1 to , and if the penalty i  is chosen properly, an infinite pool of hidden units can be achieved. A way to parameterize i  is suggested in [14], which is…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On better training the infinite restricted Boltzmann machines

Peng

Gao

2018

Mach Learn

View full text Add to dashboard Cite

The infinite restricted Boltzmann machine (iRBM) is an extension of the classic RBM. It enjoys a good property of automatically deciding the size of the hidden layer according to specific training data. With sufficient training, the iRBM can achieve a competitive performance with that of the classic RBM. However, the convergence of learning the iRBM is slow, due to the fact that the iRBM is sensitive to the ordering of its hidden units, the learned filters change slowly from the left-most hidden unit to right. To break this dependency between neighboring hidden units and speed up the convergence of training, a novel training strategy is proposed. The key idea of the proposed training strategy is randomly regrouping the hidden units before each gradient descent step. Potentially, a mixing of infinite many iRBMs with different permutations of the hidden units can be achieved by this learning method, which has a similar effect of preventing the model from over-fitting as the dropout. The original iRBM is also modified to be capable of carrying out discriminative training. To evaluate the impact of our method on convergence speed of learning and the model's generalization ability, several experiments have been performed on the binarized MNIST and CalTech101 Silhouettes datasets. Experimental results indicate that the proposed training strategy can greatly accelerate learning and enhance generalization ability of iRBMs.

show abstract

Generative and discriminative infinite restricted Boltzmann machine training

Wang

Gao

Wan

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

As one of the essential deep learning models, a restricted Boltzmann machine (RBM) is a commonly used generative training model. By adaptively growing the size of the hidden units, infinite RBM (IRBM) is obtained, which possesses an excellent property of automatically choosing the hidden layer size depending on a specific task. An IRBM presents a competitive generative capability with the traditional RBM. First, a generative model called Gaussian IRBM (GIRBM) is proposed to deal with practical scenarios from the perspective of data discretization. Subsequently, a discriminative IRBM (DIRBM) and a discriminative GIRBM (DGIRBM) are established to solve classification problems by attaching extra-label units next to the input layer. They are motivated by the fact that a discriminative variant of an RBM can complete an individual framework for classification with better performance than some standard classifiers. Remarkably, the proposed models retain both generative and discriminative properties synchronously, that is, they can reconstruct data effectively and be established in considerable self-contained classifiers. The experimental results on image classification (both large and small), text identification, and facial recognition (both clean and noisy) reflect that a DIRBM and a DGIRBM are superior to some state-of-the-art RBM models in

show abstract

An Infinite Restricted Boltzmann Machine

Cited by 52 publications

References 10 publications

A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data

A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data

On better training the infinite restricted Boltzmann machines

Generative and discriminative infinite restricted Boltzmann machine training

Contact Info

Product

Resources

About