Characterizing the sample complexity of private learners

Abstract

In 2008, Kasiviswanathan et al. defined private learning as a combination of PAC learning and differential privacy [17]. Informally, a private learner is applied to a collection of labeled individual information and outputs a hypothesis while preserving the privacy of each individual. Kasiviswanathan et al. gave a generic construction of private learners for (finite) concept classes, with sample complexity logarithmic in the size of the concept class. This sample complexity is higher than what is needed for non-private learners, hence leaving open the possibility that the sample complexity of private learning may be sometimes significantly higher than that of non-private learning.We give a combinatorial characterization of the sample size sufficient and necessary to privately learn a class of concepts. This characterization is analogous to the well known characterization of the sample complexity of non-private learning in terms of the VC dimension of the concept class. We introduce the notion of probabilistic representation of a concept class, and our new complexity measure RepDim corresponds to the size of the smallest probabilistic representation of the concept class.We show that any private learning algorithm for a concept class C with sample complexity m implies RepDim(C) = O(m), and that there exists a private learning algorithm with sample complexity m = O(RepDim(C)). We further demonstrate that a similar characterization holds for the database size needed for privately computing a large class of optimization problems and also for the well studied problem of private data release.

The structure of optimal private tests for simple hypotheses

“…There are also several rich lines of work attempting to give tight instance-specific characterizations of the sample complexities of various differentially private computations, most notably linear query release [HT10, BDKT12, NTZ16, Nik15, KN16] and PAC and agnostic learning [KLN + 08, BNS13,FX14]. The problems considered in these works are arguably more complex than the hypothesis testing problems we consider here, the characterizations are considerably looser, and are only optimal up to polynomial factors.…”

Untitled

“…They also showed that there exists an improper ε-private learner for this class, with sample complexity O α,β ,ε (1). An alternative private learner for this class was presented in [4].…”

“…There exists an efficient (α, β , ε, δ , m)-PPAC proper-learner for RECTANGLE n d , where This should be contrasted with θ α,β (n), which is the non-private sample complexity for this class (as the VC-dimension of RECTANGLE n d is 2n), and with θ α,β ,ε (nd) which is the pure-private sample complexity for this class. 4 …”

“…A complete characterization for the sample complexity of pure-private learners was recently given in [4], in terms of a new dimension-the Representation Dimension, that is, given a class C, the number of samples needed and sufficient for privately learning C is Θ(RepDim(C)). Following that, Feldman and Xiao [17] showed an equivalence between the representation dimension of a concept C and the randomized one-way communication complexity of the evaluation problem for concepts from C. Using this equivalence they separated the sample complexity of pure-private learners from that of non-private ones.…”

Private PAC learning implies finite Littlestone dimension

“…Therefore, it will be interesting to find a natural invariant that characterizes which classes can be learned privately (like the way the VC dimension characterizes PAC learning [Blumer et al, 1989, Vapnik andChervonenkis, 1971]). Such parameters exist in the case of pure differentially private learning; these include the one-way communication complexity characterization by Feldman and Xiao [2015] and the representation dimension by Beimel et al [2013]. However, no such parameter is known for approximate differentially private learning.…”