Unsupervised Learning of Spatiotemporally Coherent Metrics

Abstract

Current state-of-the-art classification and detection algorithms train deep convolutional networks using labeled data. In this work we study unsupervised feature learning with convolutional networks in the context of temporally coherent unlabeled data. We focus on feature learning from unlabeled video data, using the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity priors. We establish a connection between slow feature learning and metric learning. Using this connection we define "temporal coherence"-a criterion which can be used to set hyper-parameters in a principled and automated manner. In a transfer learning experiment, we show that the resulting encoder can be used to define a more semantically coherent metric without the use of labels.

Exploring a Mixed Representation for Encoding Temporal Coherence

“…When P = 0 the network is an invariance-free sparse autoencoder with no additional temporal cost. When P = N , temporal coherence is encoded in every hidden layer neuron, similar to the architecture described in [6]. With values inbetween, a mixed representation of invariant and variable features forms.…”

“…This method has improved object identity performance on the benchmark COIL100 dataset [11], applying the temporal coherence to the output of a Convolutional Neural Network [12], alongside sparsity in a deep invariant architecture [13] and during pre-training and network output regularization [14]. An architecture relatively similar in nature to ours is that used by Goroshin et al [6]. Temporal coherence is applied to sparse autoencoders, as part of a convolutional architecture.…”

“…Section 3 introduces a mixed autoencoder, based on a commonly used method for discovering the important variance underlying visual data, the sparse autoencoder [5]. Sparse autoencoders have previously been adapted to encode temporal coherence [6], with the extra regularization enforced across every feature. We trained a range of models on three video datasets, applying temporal coherence regularisation over different sized subsets of the total available features.…”

Exploiting Spatio-Temporal Structure With Recurrent Winner-Take-All Networks

“…al. [3] Wang and Gupta [31]. Their approach was based on metric learning of related frames in video, but their approaches were not capable of learning long term dependencies since they assumed only a simple zeromean Gaussian innovation between frames.…”

“…A machine learning application of this paradigm is to use temporal coherence as a proxy for learning sensory representations without strong supervision or explicit labels [3][4] [5]. One approach is a Bayesian formulation in which we assume the learning system builds an internal model of the world p(x t ; θ) for explaining input streams x t using a system parameterized by θ. Predictive Coding proposes to adapt this model to reduce discrepancies between predictionsp(x t |x t−1 ; θ) and observations p(x t ).…”