On the Value of Partial Information for Learning from Examples

Abstract

The PAC model of learning and its extension to real valued function classes provides a well-accepted theoretical framework for representing the problem of learning a target function g(x) using a random sample {(Based on the uniform strong law of large numbers the PAC model establishes the sample complexity, i.e., the sample size m which is sufficient for accurately estimating the target function to within high confidence. Often, in addition to a random sample, some form of prior knowledge is available about the target. It is intuitive that increasing the amount of information should have the same effect on the error as increasing the sample size. But quantitatively how does the rate of error with respect to increasing information compare to the rate of error with increasing sample size? To answer this we consider a new approach based on a combination of information-based complexity of Traub et al. and Vapnik-Chervonenkis (VC) theory. In contrast to VC-theory where function classes of finite pseudo-dimension are used only for statistical-based estimation, we let such classes play a dual role of functional estimation as well as approximation. This is captured in a newly introduced quantity, ρ d (F ), which represents a nonlinear width of a function class F. We then extend the notion of the nth minimal radius of information and define a quantity I n, d (F ) which measures the minimal approximation error of the worst-case target g ∈ F by the family of function classes having pseudo-dimension d given partial information on g consisting of values taken by n linear operators. The error rates are calculated which leads to a quantitative notion of the value of partial information for the paradigm of learning from examples.

Learning with side information: PAC learning bounds

“…For dependent data see also the paper by Campi and Kumar [5]. For related learning problems see also Blum and Mitchell [4] and learning with hints by Abu-Mostafa [1][2][3] as well as more theoretical work by Ratsaby and Maiorov [15]. As the focus of this paper is the exponential term in the uniform convergence rate we share similar aims with Vayatis [18] and Vayatis and Azencott [20], who has studied distribution dependent Vapnik-Chervonenkis bounds.…”

Approximation in shift-invariant spaces with deep ReLU neural networks

“…The n-width ρ n was firstly introduced by Maiorov and Ratsaby [22,30]. They also gave upper and lower estimates of the n-width for Sobolev spaces.…”

Continuous Algorithms in n-Term Approximation and Non-Linear Widths

“…The asymptotic order (13) has been proven in [6] for a n and $ n . The upper bound of (12) (13) is given by a continuous non-linear algorithm of approximation by the dictionary V which is constructed as follows.…”

“…Our attention is primarily focused on continuous algorithms in n-term approximation and the relevant non-linear n-widths : n , { n , {$ n for classes of functions with common mixed smoothness. Interesting ideas concerning non-linear n-widths, which are not based on continuous algorithms, have been recently introduced in [13] and [16]. For other notions of non-linear n-widths, see [18], [3].…”