Learning Poisson Binomial Distributions

Abstract

We consider a basic problem in unsupervised learning: learning an unknown Poisson binomial distribution. A Poisson binomial distribution (PBD) over {0, 1, . . . , n} is the distribution of a sum of n independent Bernoulli random variables which may have arbitrary, potentially non-equal, expectations. These distributions were first studied by Poisson (Recherches sur la Probabilitè des jugements en matié criminelle et en matiére civile. Bachelier, Paris, 1837) and are a natural n-parameter generalization of the familiar Binomial Distribution. Surprisingly, prior to our work this basic learning problem was poorly understood, and known results for it were far from optimal. We essentially settle the complexity of the learning problem for this basic class of distributions. As our first main result we give a highly efficient algorithm which learns to -accuracy (with respect to the total variation distance) usingÕ(1/ 3 ) samples independent of n. The running time of the algorithm is quasilinear in the size 123 Algorithmica of its input data, i.e.,Õ(log(n)/ 3 ) bit-operations (we writeÕ(·) to hide factors which are polylogarithmic in the argument toÕ(·); thus, for example,Õ(a log b) denotes a quantity which is O (a log b · log c (a log b)) for some absolute constant c. Observe that each draw from the distribution is a log(n)-bit string). Our second main result is a proper learning algorithm that learns to -accuracy usingÕ(1/ 2 ) samples, and runs in time (1/ ) poly(log(1/ )) · log n. This sample complexity is nearly optimal, since any algorithm for this problem must use (1/ 2 ) samples. We also give positive and negative results for some extensions of this learning problem to weighted sums of independent Bernoulli random variables.

Beyond Trace Reconstruction: Population Recovery from the Deletion Channel

“…We close this subsection by observing that while the results of [Roo00, DP15] were used in a crucial way in subsequent work of Daskalakis et al [DDS15] to obtain a sample complexity upper bound on learning Poisson binomial distributions, in our context we use these results to obtain a sample complexity lower bound for population recovery. Intuitively, the difference is that in the [DDS15] scenario of learning an unknown Poisson binomial distribution, there is no noise process affecting the samples: the learning algorithm is assumed to directly receive draws from the underlying Poisson binomial distribution being learned.…”

Learning Sums of Independent Random Variables with Sparse Collective Support

“…Consider the following very simple learning problem: Let {X i } n i=1 be independent random variables where X i is promised to be supported on the two-element set {0, i} but Pr[X i = i] is unknown: what is the sample complexity of learning X = X 1 + · · · + X N ? Even though each random variable X i is "as simple as a non-trivial random variable can be" -supported on just two values, one of which is zero -a straightforward lower bound given in [DDS12b] shows that any algorithm for learning X even to constant accuracy must use Ω(N ) samples, which is not much better than the trivial brute-force algorithm based on support size.…”

“…the union supp(X 1 ) ∪ · · · ∪ supp(X N ) of their support sets is small. Inspired by this, Daskalakis et al [DDS12b] studied the simplest non-trivial version of this learning problem, in which each X i is a Bernoulli random variable (so the union of all supports is simply {0, 1}; note, though, that the X i 's may have distinct and arbitrary biases). The main result of [DDS12b] is that this class (known as Poisson Binomial Distributions) can be learned to error ε with poly(1/ε) samples -so, perhaps unexpectedly, the complexity of learning this class is completely independent of N , the number of summands.…”

“…Inspired by this, Daskalakis et al [DDS12b] studied the simplest non-trivial version of this learning problem, in which each X i is a Bernoulli random variable (so the union of all supports is simply {0, 1}; note, though, that the X i 's may have distinct and arbitrary biases). The main result of [DDS12b] is that this class (known as Poisson Binomial Distributions) can be learned to error ε with poly(1/ε) samples -so, perhaps unexpectedly, the complexity of learning this class is completely independent of N , the number of summands. The proof in [DDS12b] relies on several sophisticated results from probability theory, including a discrete central limit theorem from [CGS11] (proved using Stein's method) and a "moment matching" result due to Roos [Roo00].…”

“…The main result of [DDS12b] is that this class (known as Poisson Binomial Distributions) can be learned to error ε with poly(1/ε) samples -so, perhaps unexpectedly, the complexity of learning this class is completely independent of N , the number of summands. The proof in [DDS12b] relies on several sophisticated results from probability theory, including a discrete central limit theorem from [CGS11] (proved using Stein's method) and a "moment matching" result due to Roos [Roo00]. (A subsequent sharpening of the [DDS12b] result in [DKS16b], giving improved time and sample complexities, also employed sophisticated tools, namely Fourier analysis and algebraic geometry.)…”