A sequential dictionary learning algorithm with enforced sparsity

Abstract

Dictionary learning algorithms have received widespread ac ceptance when it comes to data analysis and signal represen tations problems. These algorithms alternate between two stages: the sparse coding stage and dictionary update stage. In all existing dictionary learning algorithms the use of spar sity has been limited to the sparse coding stage while pre senting differences in the dictionary update stage which can be achieved sequentially or in parallel. The singular value decomposition (SVD) has been successfully used for sequen tial dictionary update. In this paper we propose a dictionary learning algorithm that include a sparsity constraint also in the dictionary update stage. The cost function used to include sparsity in the dictionary update stage is derived using the link between SVD and rank one matrix approximation. The effec tiveness of the proposed dictionary learning method is tested on synthetic data and an image processing application. The results reveal that including a sparsity constraint in the dictio nary update stage is not a bad idea.Index Terms-Dictionary learning, sparsity, SVD, se quential update, penalized rank one approximation. 1. IN TRODUCTION Dictionary learning methods have been successfully used in a number of signal and image processing applications among them image denoising [1][2], face recognition [3], compres sion [4] and fMRI data analysis [S][6][7]. These methods seek to uncover a linear multivariable latent structure in the observed data by imposing the reasonable constraint that the estimates of the observed variables are sparse linear func tions of some unknown regressors (atoms of the dictionary). The other particularity of these methods is that this set of regressors or the dictionary is also tuned iteratively to find the optimal linear multivariable latent model. Dictionary learning algorithms consist of two stages: a sparse coding stage and a dictionary update stage. In the first stage the dictionary is kept constant and the sparsity assumption is used to produce sparse linear approximations of the ob served data. In the second stage, the coefficients of the linear combination are kept constant and the dictionary is updated to minimize a certain cost function. The dictionary learning methods iterate between these two stages until convergence. The perfonnance of these methods strongly depend on the dictionary update stage since most of these methods share a similar sparse coding stage. Dictionary learning through the dictionary update stage can also be made sequential or paral lel. For a specific strategy or goal (characterized by the cost function used to update the dictionary), the parallel approach will generally require a lower computational cost than the sequential approach that uses the same strategy. While the parallel update approach may be preferred for its advantage in computational complexity, the sequential approach generally offers better results because it generates finer tuned dictionary atoms. Probabilistic and non-probabilistic approaches have been adop...

An $\alpha$ -Divergence-Based Approach for Robust Dictionary Learning

“…In particular, the function used in the fidelity term belongs to the class of redescending M-estimators which guarantee stability of inference for relatively large deviations from the nominal noise model [24]. The approach adopted for deriving the proposed algorithm exploits the observation that the observed data matrix can be approximated as a sum of rank-1 matrix approximation [25]- [27]. The proposed algorithm is obtained via adaptive sequential penalized rank-1 matrix approximation where a block coordinate descent approach is used to determine the unknowns of the different rank-1 approximation matrices.…”

“…where d j and x i denote the j th column of the dictionary D and i th column of the sparse code matrix X. Variants of (1) have also been used to design DL algorithms, among these, one can cite methods that replace the 0 −norm sparsity constraint by an 1 −norm penalty for sparsity [25], [26], [28], [30]- [33] or includes additional properties on the dictionary such as incoherence [34], [35].…”

“…DL algorithms of this form solve (1) or its variants typically by alternating between the sparse coding stage and the dictionary update stage until convergence. Another approach recently adopted in [25]- [28] for DL methods consists in approximating the dataset Y by a penalized sum of rank-1 matrices and uses a simple and exact block coordinate descent approach to estimate the factors of various rank-1 matrices. This approach can be seen as an extension of the power method or alternating least square method [40] for computing the rank-1 matrix approximation generated by the SVD.…”

“…This approach can be seen as an extension of the power method or alternating least square method [40] for computing the rank-1 matrix approximation generated by the SVD. This is the approach adopted in [25]- [28] and it works directly on E j = Y − K i=1,i =j d i x i to generate new pairs (d j , x j ) using the objective…”

Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) and Its Application to Inverse Problems

“…Some of these algorithms (e.g., [30], [38], [40]) also partially update X in the dictionary update step. A few recent methods update D and X jointly in an iterative fashion [42], [43].…”