Building high-level features using large scale unsupervised learning

Abstract

We consider the problem of building highlevel, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 22,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.

Real-time Hebbian Learning from Autoencoder Features for Control Tasks

“…ANNs of many hidden layers) through a pretraining phase in which a stack of autoencoders is trained in sequence, each one from the previous (Bengio et al, 2007;Hinton et al, 2006;Le et al, 2012;Marc'Aurelio et al, 2007), leading to a hierarchy of increasingly high-level features. Because it was thought that backpropagation struggles to train networks of many layers directly, pre-training a stack of such autoencoders and then later completing training through e.g.…”

“…Another appeal of autoencoders is that there are many ways to train them and many tricks to encourage them to produce meaningful features (Ranzato et al, 2006;Le et al, 2012). While RBMs (a kind of probabilistic model) can play a similar role to autoencoders, classic autoencoders in deep learning are generally trained through some form of stochastic gradient descent (Le et al, 2012;Bengio et al, 2007) (like backpropagation), as is the case in this paper.…”

“…While RBMs (a kind of probabilistic model) can play a similar role to autoencoders, classic autoencoders in deep learning are generally trained through some form of stochastic gradient descent (Le et al, 2012;Bengio et al, 2007) (like backpropagation), as is the case in this paper. However, the important issue in the present investigation is not the particular details of the autoencoder; in fact an advantage of the RAAHN formulation is that any autoencoder can be plugged into the RAAHN.…”

“…Interestingly, such a feature-generating system already exists and in fact has become quite popular through the rise of deep learning (Bengio et al, 2007;Hinton et al, 2006;Le et al, 2012;Marc'Aurelio et al, 2007): the autoencoder (Hinton and Zemel, 1994;Bourlard and Kamp, 1988). Autoencoders, which can be trained through a variety of algorithms from Restricted Boltzmann Machines (RBMs) (Hinton et al, 2006) to more conventional stochastic gradient descent (Le et al, 2012) (e.g.…”

“…Autoencoders, which can be trained through a variety of algorithms from Restricted Boltzmann Machines (RBMs) (Hinton et al, 2006) to more conventional stochastic gradient descent (Le et al, 2012) (e.g. similar to backpropagation; Rumelhart et al 1986), aim simply to output the same pattern as they input.…”