Teaching and Compressing for Low VC-Dimension

Abstract

In this work we study the quantitative relation between VC-dimension and two other basic parameters related to learning and teaching. Namely, the quality of sample compression schemes and of teaching sets for classes of low VC-dimension. Let C be a binary concept class of size m and VC-dimension d. Prior to this work, the best known upper bounds for both parameters were log(m), while the best lower bounds are linear in d. We present significantly better upper bounds on both as follows. Set k = O(d2 d log log |C|).We show that there always exists a concept c in C with a teaching set (i.e. a list of c-labeled examples uniquely identifying c in C) of size k. This problem was studied by Kuhlmann (1999). Our construction implies that the recursive teaching (RT) dimension of C is at most k as well. The RT-dimension was suggested by Zilles et al. and Doliwa et al. (2010). The same notion (under the name partial-ID width) was independently studied by Wigderson and Yehudayoff (2013). An upper bound on this parameter that depends only on d is known just for the very simple case d = 1, and is open even for d = 2. We also make small progress towards this seemingly modest goal.We further construct sample compression schemes of size k for C, with additional information of k log(k) bits. Roughly speaking, given any list of C-labelled examples of arbitrary length, we can retain only k labeled examples in a way that allows to recover the labels of all others examples in the list, using additional k log(k) information bits. This problem was first suggested by Littlestone and Warmuth (1986). * A preliminary version of this work, combined with the paper "Sample compression schemes for VC classes" by the first and the last authors, was published in the proceeding of FOCS'15.

Labeled Compression Schemes for Extremal Classes

“…Thus, it suffices to consider only finite concept classes. Floyd and Warmuth (1995) constructed sample compression schemes of size log |C| for every concept class C. More recently Moran et al (2015) have constructed sample compression schemes of size exp(d) log log |C| where d = V Cdim(C). Finally, Moran and Yehudayoff (2016) have constructed sample compression scheme of size exp(d), resolving Littlestone and Warmuth's question.…”

Preference-based Teaching

“…Proof. We first show that there exists a class of VC-dimension 1, say L ∞ , such that It was observed by [12] already that RTD(L ∞ ) = ∞ because every teaching set for some [0, a] must contain an infinite sequence of distinct reals that converges from above to a. Thus, using Equation (5) with…”

“…, g * ℓ * ). Let the student prefer G over G if any of the following conditions is satisfied: where with h given by (12). Let G ′ = G \ G * be the set of redundant generators in G and note that…”

“…The model of recursive teaching [18,11], which is free of coding tricks according to the Goldman-Mathias definition, has recently gained attention because its complexity parameter, the recursive teaching dimension (RTD), has shown relations to the VCdimension and to sample compression [3,4,12,16], when focusing on finite concept classes. Below though we will give examples of rather simple infinite concept classes with infinite RTD, suggesting that the RTD is inadequate for addressing the complexity of teaching infinite classes.…”

Optimal Collusion-Free Teaching

“…Models of machine learning from carefully chosen examples, i.e., from teachers, have gained increased interest in recent years, due to various application areas, such as robotics (Argall et al, 2009), trustworthy AI (Zhu et al, 2018), and pedagogy (Shafto et al, 2014). Machine teaching is also related to inverse reinforcement learning (Ho et al, 2016), to sample compression (Moran et al, 2015;Doliwa et al, 2014), and to curriculum learning (Bengio et al, 2009). The paper at hand is concerned with abstract notions of teaching, as studied in computational learning theory.…”