Topics and Techniques in Distribution Testing

Canonne, Clément L.

doi:10.1561/9781638281016

Cited by 3 publications

(1 citation statement)

References 20 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, works originating from the field of property testing in computer science [8] focused on testing uniformity against discrete distributions that do not necessarily arise as binned versions of smooth densities [5], [6], [9], [10], [11]. Instead, they may be unrestricted or obey other properties [12], [13], [14]. Furthermore, the focus is usually on the case of n much smaller than N , denote as the "sub-linear" regime.…”

Section: A Backgroundmentioning

confidence: 99%

Two-sample Testing of Discrete Distributions under Rare/Weak Perturbations

Kipnis

Donoho

2021

2021 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We consider the problem of testing the fit of a discrete sample of items from many categories to the uniform distribution over the categories. As a class of alternative hypotheses, we consider the removal of an ℓ p ball of radius ϵ around the uniform rate sequence for p ≤ 2. We deliver a sharp characterization of the asymptotic minimax risk when ϵ → 0 as the number of samples and number of dimensions go to infinity, for testing based on the occurrences' histogram (number of absent categories, singletons, collisions, ...). For example, for p = 1 and in the limit of a small expected number of samples n compared to the number of categories N (aka "sub-linear" regime), the minimax risk R * ϵ asymptotes to 2 Φ nϵ 2 / √ 8N , with Φ(x) the normal survival function. Empirical studies over a range of problem parameters show that this estimate is accurate in finite samples, and that our test is significantly better than the chisquared test or a test that only uses collisions. Our analysis is based on the asymptotic normality of histogram ordinates, the equivalence between the minimax setting to a Bayesian one, and the reduction of a multi-dimensional optimization problem to a one-dimensional problem.

show abstract

Section: A Backgroundmentioning

confidence: 99%

Two-sample Testing of Discrete Distributions under Rare/Weak Perturbations

Kipnis

Donoho

2021

2021 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

Better Sum Estimation via Weighted Sampling

Beretta,

Tětek

2024

ACM Trans. Algorithms

View full text Add to dashboard Cite

Given a large set U where each item a ∈ U has weight w ( a ), we want to estimate the total weight W = ∑ a ∈ U w ( a ) to within factor of 1 ± ε with some constant probability > 1/2. Since n = | U | is large, we want to do this without looking at the entire set U . In the traditional setting in which we are allowed to sample elements from U uniformly, sampling Ω ( n ) items is necessary to provide any non-trivial guarantee on the estimate. Therefore, we investigate this problem in different settings: in the proportional setting we can sample items with probabilities proportional to their weights, and in the hybrid setting we can sample both proportionally and uniformly. These settings have applications, for example, in sublinear-time algorithms and distribution testing. Sum estimation in the proportional and hybrid setting has been considered before by Motwani, Panigrahy, and Xu [ICALP, 2007]. In their paper, they give both upper and lower bounds in terms of n . Their bounds are near-matching in terms of n , but not in terms of ε. In this paper, we improve both their upper and lower bounds. Our bounds are matching up to constant factors in both settings, in terms of both n and ε. No lower bounds with dependency on ε were known previously. In the proportional setting, we improve their \(\tilde{O}(\sqrt {n}/\varepsilon ^{7/2}) \) algorithm to \(O(\sqrt {n}/\varepsilon) \) . In the hybrid setting, we improve \(\tilde{O}(\sqrt [3]{n}/ \varepsilon ^{9/2}) \) to \(O(\sqrt [3]{n}/\varepsilon ^{4/3}) \) . Our algorithms are also significantly simpler and do not have large constant factors. We then investigate the previously unexplored scenario in which n is not known to the algorithm. In this case, we obtain a \(O(\sqrt {n}/\varepsilon + \log n / \varepsilon ^2) \) algorithm for the proportional setting, and a \(O(\sqrt {n}/\varepsilon) \) algorithm for the hybrid setting. This means that in the proportional setting, we may remove the need for advice without greatly increasing the complexity of the problem, while there is a major difference in the hybrid setting. We prove that this difference in the hybrid setting is necessary, by showing a matching lower bound. Our algorithms have applications in the area of sublinear-time graph algorithms. Consider a large graph G = ( V , E ) and the task of (1 ± ε)-approximating | E |. We consider the (standard) settings where we can sample uniformly from E or from both E and V . This relates to sum estimation as follows: we set U = V and the weights to be equal to the degrees. Uniform sampling then corresponds to sampling vertices uniformly. Proportional sampling can be simulated by taking a random edge and picking one of its endpoints at random. If we can only sample uniformly from E , then our results immediately give a \(O(\sqrt {|V|} / \varepsilon) \) algorithm. When we may sample both from E and V , our results imply an algorithm with complexity \(O(\sqrt [3]{|V|}/\varepsilon ^{4/3}) \) . Surprisingly, one of our subroutines provides an (1 ± ε)-approximation of | E | using \(\tilde{O}(d/\varepsilon ^2) \) expected samples, where d is the average degree, under the mild assumption that at least a constant fraction of vertices are non-isolated. This subroutine works in the setting where we can sample uniformly from both V and E . We find this remarkable since it is O (1/ε 2 ) for sparse graphs.

show abstract