Rademacher complexity of margin multi-category classifiers

Abstract

One of the main open problems of the theory of margin multicategory pattern classification is the characterization of the optimal dependence of the confidence interval of a guaranteed risk on the three basic parameters which are the sample size m, the number C of categories and the scale parameter γ. This is especially the case when working under minimal learnability hypotheses. The starting point is a basic supremum inequality whose capacity measure depends on the choice of the margin loss function. Then, transitions are made, from capacity measure to capacity measure. At some level, a structural result performs the transition from the multi-class case to the bi-class one. In this article, we highlight the advantages and drawbacks inherent to the three major options for this decomposition: using Rademacher complexities, covering numbers or scale-sensitive combinatorial dimensions.

Multi-category classifiers and sample width

“…The bound of Theorem 5.2 compares well with the margin-based bound from [11]. It varies as 1/ √ m, while that of [11] has an additional √ ln m factor.…”

“…The dependence on C is, however, √ C whereas in [11] it is √ ln 2 C. The bounds are not directly comparable because ours concerns width and that of [11] involves margin, but, for fixed C, it is notable that the m-dependence of our bound is better. The dependence of Theorem 5.2 on the width parameter γ is, in general, similar to the dependence of [11] on the margin parameter γ as both grow like √ N γ where N γ is the covering number of the underlying realvalued discriminant function class. The advantage of the bound of Theorem 5.2 is that it is expressed in terms of the covering number of the actual metric space which, in some problems, such as when the metric space is finite [5], can be efficiently estimated.…”

“…Margin-based results apply when the binary classifiers are derived from realvalued function by 'thresholding' (taking their sign). Margin analysis has been extended to multi-category classifiers in [11].…”

“…It varies as 1/ √ m, while that of [11] has an additional √ ln m factor. The dependence on C is, however, √ C whereas in [11] it is √ ln 2 C. The bounds are not directly comparable because ours concerns width and that of [11] involves margin, but, for fixed C, it is notable that the m-dependence of our bound is better.…”

“…It varies as 1/ √ m, while that of [11] has an additional √ ln m factor. The dependence on C is, however, √ C whereas in [11] it is √ ln 2 C. The bounds are not directly comparable because ours concerns width and that of [11] involves margin, but, for fixed C, it is notable that the m-dependence of our bound is better. The dependence of Theorem 5.2 on the width parameter γ is, in general, similar to the dependence of [11] on the margin parameter γ as both grow like √ N γ where N γ is the covering number of the underlying realvalued discriminant function class.…”