Generalized Mercer Kernels and Reproducing Kernel Banach Spaces

Abstract

This article studies constructions of reproducing kernel Banach spaces (RKBSs) which may be viewed as a generalization of reproducing kernel Hilbert spaces (RKHSs). A key point is to endow Banach spaces with reproducing kernels such that machine learning in RKBSs can be well-posed and of easy implementation. First we verify many advanced properties of the general RKBSs such as density, continuity, separability, implicit representation, imbedding, compactness, representer theorem for learning methods, oracle inequality, and universal approximation. Then, we develop a new concept of generalized Mercer kernels to construct p-norm RKBSs for 1 ≤ p ≤ ∞. The p-norm RKBSs preserve the same simple format as the Mercer representation of RKHSs. Moreover, the p-norm RKBSs are isometrically equivalent to the standard p-norm spaces of countable sequences. Hence, the pnorm RKBSs possess more geometrical structures than RKHSs including sparsity. To be more precise, the suitable countable expansion terms of the generalized Mercer kernels can be used to represent the pairs of Schauder bases and biorthogonal systems of the p-norm RKBSs such that the generalized Mercer kernels become the reproducing kernels of the p-norm RKBSs. The generalized Mercer kernels also cover many well-known kernels, for example, min kernels, Gaussian kernels, and power series kernels. Finally, we propose to solve the support vector machines in the p-norm RKBSs, which are to minimize the regularized empirical risks over the p-norm RKBSs. We show that the infinite dimensional support vector machines in the p-norm RKBSs can be equivalently transferred to finite dimensional convex optimization problems such that we obtain the finite dimensional representations of the support vector machine solutions for practical applications. In particular, we verify that some special support vector machines in the 1-norm RKBSs are equivalent to the classical 1-norm sparse regressions. This gives fundamental supports of a novel learning tool called sparse learning methods to be investigated in our next research project.

A Unifying Representer Theorem for Inverse Problems and Machine Learning

“…The concept of reproducing kernel Banach space, which is the natural generalization of RKHS, was introduced and investigated by Zhang and Xu in [55,56]. Similar to the Hilbertian case, one can identify the RKBS property as follows.…”

“…This theorem, in its extended version [42], is the foundation for the majority of kernelbased methods for machine learning, including regression, radial-basis functions, and support-vector machines [23,44,48]. There is also a whole line of generalizations of the concept that involves reproducing kernel Banach spaces (RKBS) [55][56][57]. More recently, motivated by the success of 1 and total-variation regularization for compressed sensing [11,14,19], researchers have derived alternative representer theorems in order to explain the sparsifying effect of such penalties and their robustness to missing data [8,26,28,52].…”

Sparse regularized learning in the reproducing kernel banach spaces with the <inline-formula><tex-math id="M1">\begin{document}$ \ell^1 $\end{document}</tex-math></inline-formula> norm

“…The notion of RKBS was originally introduced by Zhang, Xu, and Zhang [25] in 2009 for machine learning, which serves as a generalization of RKHS. Since then, there has been emerging interest in machine learning in various RKBSs [14,20,24]. Notice that the reproducing kernel of an RKBS is not necessarily unique or positive definite [25,24].…”

“…Since then, there has been emerging interest in machine learning in various RKBSs [14,20,24]. Notice that the reproducing kernel of an RKBS is not necessarily unique or positive definite [25,24]. The sparsity-promoting regularization networks in the 1 -norm RKBSs are given by…”

“…Additional conditions are required to guarantee its existence. Representer theorems for various reflexive RKBSs were developed in the literature [14,24,25]. Note that the RKBS B K with the 1 norm is nonreflexive.…”