Role of Homeostasis in Learning Sparse Representations

Abstract

Neurons in the input layer of primary visual cortex in primates develop edge-like receptive fields. One approach to understanding the emergence of this response is to state that neural activity has to efficiently represent sensory data with respect to the statistics of natural scenes. Furthermore, it is believed that such an efficient coding is achieved using a competition across neurons so as to generate a sparse representation, that is, where a relatively small number of neurons are simultaneously active. Indeed, different models of sparse coding, coupled with Hebbian learning and homeostasis, have been proposed that successfully match the observed emergent response. However, the specific role of homeostasis in learning such sparse representations is still largely unknown. By quantitatively assessing the efficiency of the neural representation during learning, we derive a cooperative homeostasis mechanism that optimally tunes the competition between neurons within the sparse coding algorithm. We apply this homeostasis while learning small patches taken from natural images and compare its efficiency with state-of-the-art algorithms. Results show that while different sparse coding algorithms give similar coding results, the homeostasis provides an optimal balance for the representation of natural images within the population of neurons. Competition in sparse coding is optimized when it is fair. By contributing to optimizing statistical competition across neurons, homeostasis is crucial in providing a more efficient solution to the emergence of independent components.

Sparse Models for Computer Vision

“…However, resulting dictionaries vary qualitatively among these schemes and it was unclear which algorithm is the most efficient and what was the individual role of the different mechanisms that constitute SHL schemes. At the learning level, we have shown that the homeostasis mechanism had a great influence on the qualitative distribution of learned filters (Perrinet, 2010).…”

“…Indeed, given a sparse coding strategy that optimizes any representation efficiency cost as defined above, we may derive an unsupervised learning model by optimizing the dictionary Φ over natural scenes. On the one hand, the flexibility in the definition of the sparseness cost leads to a wide variety of proposed sparse coding solutions (for a review, see (Pece, 2002)) such as numerical optimization (Olshausen and Field, 1997), non-negative matrix factorization (Lee and Seung, 1999;Ranzato et al, 2007) or Matching Pursuit (Perrinet et al, 2004;Smith and Lewicki, 2006;Rehn and Sommer, 2007;Perrinet, 2010). They are all derived from correlation-based inhibition since this is necessary to remove redundancies from the linear representation.…”

“…The parameters of this homeostatic rule have a great importance for the convergence of the global algorithm. In (Perrinet, 2010), we have derived a more general homeostasis mechanism derived from the optimization of the representation efficiency through histogram equalization which we will describe later (see Section 14.4.1).…”

Image Segmentation Using a Sparse Coding Model of Cortical Area V1

“…When the PC/BC algorithm (with appropriate learning rules) is trained on natural images, it learns a dictionary of basis vectors (i.e., synaptic weights) that resemble the RFs of V1 cells (Spratling, 2012b). Many other algorithms, when trained on natural images, have also been shown to be able to learn basis sets that resemble the RFs of cells in primary visual cortex (e.g., Bell and Sejnowski, 1997;Falconbridge et al, 2006;Hamker and Wiltschut, 2007;Harpur, 1997;Hoyer, 2003Hoyer, , 2004Hoyer and Hyvärinen, 2000;Jehee and Ballard, 2009;Lücke, 2009;Olshausen and Field, 1996;Perrinet, 2010;Ranzato et al, 2007;Rehn and Sommer, 2007;Van Hateren and van der Schaaf, 1998;Weber and Triesch, 2008;Wiltschut and Hamker, 2009). A common feature of all these algorithms is that the learnt representation is sparse.…”

Biologically-inspired characterization of sparseness in natural images

“…Globally, this procedure gives us a sequential algorithm for reconstructing the signal using the list of sources (filters with coefficients), which greedily optimizes the 0 pseudo-norm (i.e., achieves a relatively sparse representation given the stopping criterion). The procedure is known as the Matching Pursuit (MP) algorithm [9], which has been shown to generate good approximations for natural images [14]. For this work we made two minor improvements to this method: First, we took advantage of the response of the filters as complex numbers.…”