Sparse kernel recursive least squares using L<inf>1</inf> regularization and a fixed-point sub-iteration

Abstract

A new kernel adaptive filtering (KAF) algorithm, namely the sparse kernel recursive least squares (SKRLS), is derived by adding a l 1 -norm penalty on the center coefficients to the least squares (LS) cost (i.e. the sum of the squared errors). In each iteration, the center coefficients are updated by a fixed-point sub-iteration. Compared with the original KRLS algorithm, the proposed algorithm can produce a much sparser network, in which many coefficients are negligibly small. A much more compact structure can thus be achieved by pruning these negligible centers. Simulation results show that the SKRLS performs very well, yielding a very sparse network while preserving a desirable performance.

Kernel Recursive Least-Squares Temporal Difference Algorithms with Sparsification and Regularization

“…However, the LASSO method is inherently a batch method and is unsuitable for online learning. Instead, we resort to the fixed-point subiteration method introduced in [ 13 ]. We first use the sign function sign⁡( α t ) to replace sgn⁡( α t ) in ( 33 ).…”

“…From ( 16 ), A t −1 also requires removing some rows and columns. Unfortunately, we cannot use the method in [ 30 ] to do this like Chen et al in [ 13 ], since A t −1 is not a symmetry matrix. Considering that b t will remove the corresponding elements if 𝒟 t is pruned, we directly perform Ψ ℐ t ( A t −1 ) to remove the rows and columns indexed by ℐ t .…”

“…Thus, as long as a linear algorithm can be formulated in terms of inner products, there is no need to perform computations in the high-dimensional feature space [ 9 ]. Recently, there have also been many research works on kernelizing least-squares algorithms [ 9 – 13 ]. Here, we only review some works related to our proposed algorithms.…”

“…Besides having the good properties of SKRLS-ALD, it can avoid overfitting. In addition, Chen et al proposed an L 1 -regularized SKRLS algorithm with the fixed-point subiteration [ 13 ], which can yield a much sparser dictionary.…”

A Kernel Least Mean Square Algorithm Based on Randomized Feature Networks

“…To restrict the growth of the weight network, several online sparsification criteria were proposed for KAFs to select valid samples in the learning process, such as approximate linear dependency (ALD) [5], the novelty criterion (NC) [11], the surprise criterion (SC) [12], the coherence criterion (CC) [13], the quantization criterion [14], and sparsity-promoting regularization [15,16]. Among these criteria, the quantization criterion-based kernel least mean square (QKLMS) algorithm can perform better in the tradeoff between steady-state error and computational complexity [14].…”