Efficient sparse coding algorithms

Abstract

Sparse coding provides a class of algorithms for finding succinct representations of stimuli; given only unlabeled input data, it discovers basis functions that capture higher-level features in the data. However, finding sparse codes remains a very difficult computational problem. In this paper, we present efficient sparse coding algorithms that are based on iteratively solving two convex optimization problems: an L 1-regularized least squares problem and an L 2-constrained least squares problem. We propose novel algorithms to solve both of these optimization problems. Our algorithms result in a significant speedup for sparse coding, allowing us to learn larger sparse codes than possible with previously described algorithms. We apply these algorithms to natural images and demonstrate that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.

Predicting Brain Amyloid Using Multivariate Morphometry Statistics, Sparse Coding, and Correntropy: Validation in 1,101 Individuals From the ADNI and OASIS Databases

“…Classical dictionary learning techniques (Olshausen and Field, 1997;Lee et al, 2007) consider a finite training set of feature maps, X = (x 1 , x 2 , x n ) in R p×n . In our study, X is the set of MMS features from n surface patches of all the samples.…”

“…By defining a better lower-dimensional subspace, this information loss can be limited. Sparse coding (Lee et al, 2007;Mairal et al, 2009) has been previously proposed to learn an over-complete set of basis vectors (also called a dictionary) to represent input vectors efficiently and concisely (Donoho and Elad, 2003). Sparse coding has been shown to be effective for many tasks such as image imprinting (Moody et al, 2012), image deblurring (Yin et al, 2008), super-resolution (Yang et al, 2008), classification (Mairal et al, 2009), functional brain connectivity (Lv et al, 2015(Lv et al, , 2017, and structural morphometry analysis (Zhang et al, 2017a).…”

“…Sparse coding has been shown to be effective for many tasks such as image imprinting (Moody et al, 2012), image deblurring (Yin et al, 2008), super-resolution (Yang et al, 2008), classification (Mairal et al, 2009), functional brain connectivity (Lv et al, 2015(Lv et al, , 2017, and structural morphometry analysis (Zhang et al, 2017a). Our previous studies (Zhang et al, 2016a(Zhang et al, ,b, 2017a propose a PASS-MP method, consisting of sparse coding (Lee et al, 2007;Mairal et al, 2009), and Max-pooling (LeCun et al, 2015), for surface feature dimension reduction. PASS-MP has excellent impressive performance for the sparse coding of our MMS features.…”

Spatial position constraint for unsupervised learning of speech representations

“…PCA and factor analysis aim to find a projection that maximizes variance across data points and among the target dimensions, respectively (Ramashini et al, 2019). On the other hand, sparse coding utilizes and learns a dictionary, such that input data can be decomposed into a linear combination of a sparse code and the learned dictionary themselves (Lee et al, 2007). When projecting to a lower number of dimensions, random projection and multidimensional scaling aim at preserving geometric structure within data points (Dasgupta & Freund, 2008).…”

“…A similar effect can also be achieved by penalizing the representation layer outputs for variation in input, which is referred to as the contractive loss penalty (Rifai et al, 2011). Furthermore, self-supervised training has been applied effectively for representation learning (Raina et al, 2007). In a self-supervised setting, the auto-encoder is trained to map input data to its transformed version at the output, with the transformed output computed through manual processing techniques over the input data.…”