The Broad Optimality of Profile Maximum Likelihood

Yi, Hao; Orlitsky, Alon

doi:10.48550/arxiv.1906.03794

Cited by 7 publications

(14 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Correct asymptotic For most of the properties considered in the paper, even the naive empiricalfrequency estimator is sample-optimal in the large-sample regime (termed "simple regime" in [37]) where the number of samples n far exceeds the alphabet size k. The interesting regime, addressed in numerous recent publications [17,18,21,20,23,34,36,38], is where n and k are comparable, e.g., differing by at most a logarithmic factor. In this range, n is sufficiently small that sophisticated techniques can help, yet not too small that nothing can be estimated.…”

Section: Implications and New Resultsmentioning

confidence: 99%

“…Recently, the work of [2] showed that the profile maximum likelihood (PML) estimator [28], an estimator that finds a distribution maximizing the probability of observing the multiset of empirical frequencies, is sample-optimal for estimating entropy, distance to uniformity, and normalized support size and coverage. After the initial submission of the current work, paper [20] showed that the PML approach and its near-linear-time computable variant [5] are sample-optimal for any property that is symmetric, additive, and appropriately Lipschitz, including the four properties just mentioned. This establishes the PML estimator as the first universally sample-optimal plug-in approach for estimating symmetric properties.…”

Section: Existing Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Unified Sample-Optimal Property Estimation in Near-Linear Time

Hao,

Orlitsky

2019

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the fundamental learning problem of estimating properties of distributions over large domains. Using a novel piecewise-polynomial approximation technique, we derive the first unified methodology for constructing sample-and time-efficient estimators for all sufficiently smooth, symmetric and non-symmetric, additive properties. This technique yields near-linear-time computable estimators whose approximation values are asymptotically optimal and highly-concentrated, resulting in the first: 1) estimators achieving the O(k/(ε 2 log k)) min-max ε-error sample complexity for all k-symbol Lipschitz properties; 2) unified near-optimal differentially private estimators for a variety of properties; 3) unified estimator achieving optimal bias and near-optimal variance for five important properties; 4) near-optimal sample-complexity estimators for several important symmetric properties over both domain sizes and confidence levels. In addition, we establish a McDiarmid's inequality under Poisson sampling, which is of independent interest.

show abstract

Section: Implications and New Resultsmentioning

confidence: 99%

Section: Existing Methodsmentioning

confidence: 99%

Unified Sample-Optimal Property Estimation in Near-Linear Time

Hao,

Orlitsky

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Note the dependency on in the above theorem and the approximation factor in Theorem 3.4 are strictly better than [CSS19], which is another efficient PML based approach for universal symmetric property estimation; [CSS19] works when the error parameter ≥ 1 n 0.166 . Recent work [HO19], further gives the broad optimality of approximate PML. [HO19] shows optimality of approximate PML distribution based estimator for other symmetric properties, such as, sorted distribution estimation (under 1 distance), α-Renyi entropy for non-integer α > 3/4, and other broad class of additive properties that are Lipschitz.…”

Section: Theorem 34 (Exp (mentioning

confidence: 99%

“…Recent work [HO19], further gives the broad optimality of approximate PML. [HO19] shows optimality of approximate PML distribution based estimator for other symmetric properties, such as, sorted distribution estimation (under 1 distance), α-Renyi entropy for non-integer α > 3/4, and other broad class of additive properties that are Lipschitz. [HO19] also provides a PML-based tester to test whether an unknown distribution is ≥ far from a given distribution in 1 distance and achieves the optimal sample complexity up to logarithmic factors.…”

Section: Theorem 34 (Exp (mentioning

confidence: 99%

The Bethe and Sinkhorn Permanents of Low Rank Matrices and Implications for Profile Maximum Likelihood

Anari,

Charikar,

Shiragur

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper we consider the problem of computing the likelihood of the profile of a discrete distribution, i.e., the probability of observing the multiset of element frequencies, and computing a profile maximum likelihood (PML) distribution, i.e., a distribution with the maximum profile likelihood. For each problem we provide polynomial time algorithms that given n i.i.d. samples from a discrete distribution, achieve an approximation factor of exp (−O( √ n log n)), improving upon the previous best-known bound achievable in polynomial time of exp(−O(n 2/3 log n)) (Charikar, Shiragur and Sidford, 2019). Through the work of Acharya, Das, Orlitsky and Suresh (2016), this implies a polynomial time universal estimator for symmetric properties of discrete distributions in a broader range of error parameter.We achieve these results by providing new bounds on the quality of approximation of the Bethe and Sinkhorn permanents (Vontobel, 2012 and2014). We show that each of these are exp(O(k log(N/k))) approximations to the permanent of N × N matrices with non-negative rank at most k, improving upon the previous known bounds of exp(O(N )). To obtain our results on PML, we exploit the fact that the PML objective is proportional to the permanent of a certain Vandermonde matrix with √ n distinct columns, i.e. with non-negative rank at most √ n. As a by-product of our work we establish a surprising connection between the convex relaxation in prior work (CSS19) and the well-studied Bethe and Sinkhorn approximations.

show abstract

“…Distribution property estimation literature most related to our work include entropy estimation [19,20,21,22,23,24,25,26], support size estimation [21,23,27], Rényi entropy estimation [28,29,30], support coverage estimation [31,32], and divergence estimation [33,34].…”

Section: Introductionmentioning

confidence: 99%

Estimating Entropy of Distributions in Constant Space

Acharya,

Bhadane,

Indyk

et al. 2019

Preprint

View full text Add to dashboard Cite

We consider the task of estimating the entropy of k-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requiressamples and a constant O(1) memory words of space and outputs a ±ε estimate of H(p). Without space limitations, the sample complexity has been established as, which is sub-linear in the domain size k, and the current algorithms that achieve optimal sample complexity also require nearly-linear space in k.Our algorithm partitions [0, 1] into intervals and estimates the entropy contribution of probability values in each interval. The intervals are designed to trade off the bias and variance of these estimates.

show abstract

The Broad Optimality of Profile Maximum Likelihood

Cited by 7 publications

References 61 publications

Unified Sample-Optimal Property Estimation in Near-Linear Time

Unified Sample-Optimal Property Estimation in Near-Linear Time

The Bethe and Sinkhorn Permanents of Low Rank Matrices and Implications for Profile Maximum Likelihood

Estimating Entropy of Distributions in Constant Space

Contact Info

Product

Resources

About