On better training the infinite restricted Boltzmann machines

Abstract

The infinite restricted Boltzmann machine (iRBM) is an extension of the classic RBM. It enjoys a good property of automatically deciding the size of the hidden layer according to specific training data. With sufficient training, the iRBM can achieve a competitive performance with that of the classic RBM. However, the convergence of learning the iRBM is slow, due to the fact that the iRBM is sensitive to the ordering of its hidden units, the learned filters change slowly from the left-most hidden unit to right. To break this dependency between neighboring hidden units and speed up the convergence of training, a novel training strategy is proposed. The key idea of the proposed training strategy is randomly regrouping the hidden units before each gradient descent step. Potentially, a mixing of infinite many iRBMs with different permutations of the hidden units can be achieved by this learning method, which has a similar effect of preventing the model from over-fitting as the dropout. The original iRBM is also modified to be capable of carrying out discriminative training. To evaluate the impact of our method on convergence speed of learning and the model's generalization ability, several experiments have been performed on the binarized MNIST and CalTech101 Silhouettes datasets. Experimental results indicate that the proposed training strategy can greatly accelerate learning and enhance generalization ability of iRBMs.

An Adaptive Feature Learning Model for Sequential Radar High Resolution Range Profile Recognition

“…The second experiment investigated the robustness of the proposed model when the angular sampling rates at test phase were different from that at training phase. To accelerate the learning of Dis-iRBMs, we used a new training strategy referred as “RP” training [ 34 ] to train the Dis-iRBMs in all experiments.…”

A Multichannel Data Fusion Method Based on Multiple Deep Belief Networks for Intelligent Fault Diagnosis of Main Reducer

“…The second procedure is supervised fine-tuning of the whole networks by back propagation algorithm. In this procedure, all of the hidden layers are considered as a whole and model parameters are adjusted to decrease the training error [25].…”

Robust sequential high‐resolution range profile recognition based on the infinite conditional restricted Boltzmann machine

“…The ADAGRAD [43], a per‐dimensional learning rate was also used as [32] does. To accelerate the learning speed and achieve better generalisation, a new training strategy referred as ‘RP training’ [44] was adopted here for all the experiments.…”