On Kernels, Margins, and Low-Dimensional Mappings

Abstract

Abstract. Kernel functions are typically viewed as providing an implicit mapping of points into a high-dimensional space, with the ability to gain much of the power of that space without incurring a high cost if data is separable in that space by a large margin γ. However, the Johnson-Lindenstrauss lemma suggests that in the presence of a large margin, a kernel function can also be viewed as a mapping to a lowdimensional space, one of dimension onlyÕ(1/γ 2 ). In this paper, we explore the question of whether one can efficiently compute such implicit low-dimensional mappings, using only black-box access to a kernel function. We answer this question in the affirmative if our method is also allowed black-box access to the underlying distribution (i.e., unlabeled examples). We also give a lower bound, showing this is not possible for an arbitrary black-box kernel function, if we do not have access to the distribution. We leave open the question of whether such mappings can be found efficiently without access to the distribution for standard kernel functions such as the polynomial kernel. Our positive result can be viewed as saying that designing a good kernel function is much like designing a good feature space. Given a kernel, by running it in a black-box manner on random unlabeled examples, we can generate an explicit set ofÕ(1/γ 2 ) features, such that if the data was linearly separable with margin γ under the kernel, then it is approximately separable in this new feature space.

A discriminative framework for clustering via similarity functions

“…Some of the questions we address can be viewed as a generalization of questions studied in machine learning of what properties of similarity functions (especially kernel functions) are sufficient to allow one to learn well [6,7,21,31,29]. E.g., the usual statement is that if a kernel function satisfies the property that the target function is separable by a large margin in the implicit kernel space, then learning can be done from few labeled examples.…”

“…We also give formal relationships between these properties and those considered implicitly by approximation algorithms for standard clustering objectives. We then analyze a much weaker average-attraction property in Section 4 that has close connections to large margin properties studied in Learning Theory [6,7,21,31,29]. This property is not sufficient to produce a hierarchical clustering, however, so we then turn to the question of how weak a property can be and still be sufficient for hierarchical clustering, which leads us to analyze properties motivated by game-theoretic notions of stability in Section 5.…”

“…In particular, motivated by work on learning with kernel functions that asks "what natural properties of a given kernel (or similarity) function K are sufficient to allow one to learn well?" [6,7,31,29,21] we ask the question "what natural properties of a pairwise similarity function are sufficient to allow one to cluster well?" To study this question we develop a theoretical framework which can be thought of as a discriminative (PAC style) model for clustering, though the basic object of study, rather than a concept class, is a property of the similarity function K in relation to the target concept, or equivalently a set of (concept, similarity function) pairs.…”

Improving Random Projections Using Marginal Information

“…We will show that, from a practical point of view, using the marginal information can actually improve the bounds. 6 Since we know the exact distribution in this case, we might as well compute k exactly by iteratively solving a nonlinear equation:…”

“…Random projections [1] have been used in machine learning [2][3][4][5][6] and many other applications in data mining and information retrieval, e.g., [7][8][9][10][11][12].…”