Sparse Coding via Thresholding and Local Competition in Neural Circuits

Abstract

While evidence indicates that neural systems may be employing sparse approximations to represent sensed stimuli, the mechanisms underlying this ability are not understood. We describe a locally competitive algorithm (LCA) that solves a collection of sparse coding principles minimizing a weighted combination of mean-squared error and a coefficient cost function. LCAs are designed to be implemented in a dynamical system composed of many neuron-like elements operating in parallel. These algorithms use thresholding functions to induce local (usually one-way) inhibitory competitions between nodes to produce sparse representations. LCAs produce coefficients with sparsity levels comparable to the most popular centralized sparse coding algorithms while being readily suited for neural implementation. Additionally, LCA coefficients for video sequences demonstrate inertial properties that are both qualitatively and quantitatively more regular (i.e., smoother and more predictable) than the coefficients produced by greedy algorithms.

On rectified linear units for speech processing

“…This method can be interpreted as a special case of PSD [12,13] and sparse coding [14] when inference of features is computed in just one step. Unlike RBM's objective function which is intractable, this method can be optimized very efficiently and it enjoys the use of ReLU units because they naturally produce sparse features.…”

Sparse analog associative memory via L1-regularization and thresholding

“…as 't' varies from 0 to ∞ the minimizer of the above cost function traces through the central path that contains the unique optimal solution (x * , y * ) for (10). So, by gradually increasing the value of 't' one can update the values of (x, y) to get closer to the optimal solution.…”

Sparse coding with population sketches

“…In models for sparse coding in the primary visual cortex (V1), the receptive fields of simple cells in the input layer of V1 serve as the basis from which a sparse representation of the visual environment is constructed. Recently, a number of neurally plausible mechanisms for sparse coding were developed [1], yielding testable predictions of cortico-cortical interactions of cells within a population by inducing local competition to minimize the total number of active cells in the network. Although this algorithmic framework provides an explicit mechanism for simple cell sparse coding, many still contend that current models for sparse coding are not plausible due to the overhead required to transmit both the locations and spike rates of all active cells within the population to higher areas in the cortex.…”

“…The function of this network is still not known, however, studies of information flow in the cortex suggest that cells in these upper layers may provide a means for competition to occur amongst neighboring columns. To obtain a sketch from our model of this hypercolumn, we employed locally competitive algorithms for sparse coding [1] within each orientation minicolumn; each of these approximations are then fed forward to the fusion network which solves a sparse approximation problem to determine the minimum number of minicolumns needed to effectively represent the stimulus. We call the sparse weight vector that emerges from this computation our population sketch.…”

“…To show the viability of this approach, we considered the approximation error in light of the sparsity and the number of wires required by our model and the fully connected model in [1]. We found that for a specified approximation error, the total sparsity of the solution (total number of cells active amongst all columns) is either equal to or only slightly higher than those obtained from the fully connected network.…”