Testing Closeness of Discrete Distributions

Abstract

Given samples from two distributions over an n-element set, we wish to test whether these distributions are statistically close. We present an algorithm which uses sublinear in n, specifically, O(n 2/3 ǫ −8/3 log n), independent samples from each distribution, runs in time linear in the sample size, makes no assumptions about the structure of the distributions, and distinguishes the cases when the distance between the distributions is small (less than max{ǫ 4/3 n −1/3 /32, ǫn −1/2 /4}) or large (more than ǫ) in ℓ 1 distance. This result can be compared to the lower bound of Ω(n 2/3 ǫ −2/3 ) for this problem given by Valiant [54]. Our algorithm has applications to the problem of testing whether a given Markov process is rapidly mixing. We present sublinear algorithms for several variants of this problem as well.

Untitled

“…Batu et al [30] exhibited classical testers for this using O((m/ε) 2/3 log m) queries, 8 and Valiant [160] proved an almost matching lower bound of Ω(m 2/3 ) for constant ε. These bounds have both recently been improved by Chan et al [54] to Θ(m 2/3 /ε 4/3 ), which is tight for all ε ≥ m −1/4 .…”

“…This can be seen as the quantum generalization of the classical question of testing whether two probability distributions on d elements are equal or ε-far from equal (with respect to the total variation distance), given access to samples from the distributions. A sublinear tester for the classical problem has been given by Batu et al [30], and recently improved by Chan et al [54]; for constant ε the tester uses O(d 2/3 ) samples. By fixing σ = I/d, the result of [56] implies that the quantum generalization of this problem is more difficult: it requires at least Ω(d) "samples" (i. e., copies of the states).…”

Taming big probability distributions

“…al. [3,7], distinguishes pairs of distributions p and q that are identical from those pairs p, q which are ε-far as follows:…”

Weak Convergence Analysis of Asymptotically Optimal Hypothesis Tests

“…The objective is to determine whether or not both strings are drawn from identical distributions in P(Z). Homogeneity testing is also closely related to the problem of closeness testing [40]- [42]. As before we work in the regime where m and n are linearly related as m = λn where λ is a known constant.…”