Sparse coding with an overcomplete basis set: A strategy employed by V1?

Abstract

The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and bandpass, comparable with the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images. Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete--i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input-output function will deviate from being purely linear. These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli.

Simplicity: a unifying principle in cognitive science?

“…† The physiology of early vision, including receptive-field structures, and phenomena such as lateral inhibition, seems adapted to maximize information compression from visual input [20]. On the other hand, both theoretical and empirical arguments suggest that, the brain also uses highly redundant 'sparse' neural codes for perceptual input [21,22].…”

Multi-column deep neural network for traffic sign classification

“…Unsupervised learning methods applied to patches of natural images tend to produce localized filters that resemble off-center-on-surround filters, orientationsensitive bar detectors, Gabor filters (Schmidhuber et al, 1996;Olshausen & Field, 1997;Hoyer & Hyvärinen, 2000). These findings in conjunction with experimental studies of the visual cortex justify the use of such filters in the so-called standard model for object recognition (Riesenhuber & Poggio, 1999;Serre et al, 2005;Mutch & Lowe, 2008), whose filters are fixed, in contrast to those of Convolutional Neural Networks (CNNs) (LeCun et al, 1998;Behnke, 2003;Simard et al, 2003), whose weights (filters) are randomly initialized and learned in a supervised way using back-propagation (BP).…”

Efficient online bootstrapping of sensory representations

“…For performing the dimensionality-reducing projection step from input representations I to the induced representation N, numerous methods such as multilayer perceptrons (MLP) [18], PCA [19], sparse coding [20], k-means and indeed, SOM [16] are available. Given our long-term goals for the PROPRE algorithm, the open-ended formation of representations as envisioned in Fig.…”