Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation

Wu, Yihong; Yang, Pengkun

doi:10.1109/tit.2016.2548468

Cited by 166 publications

(300 citation statements)

References 58 publications

(48 reference statements)

Supporting

Mentioning

294

Contrasting

Order By: Relevance

“…Another estimator proposed by Valiant and Valiant [26] has only been shown to achieve the minimax rate in the restrictive regime of

\frac{S}{\ln S} ≲ n ≲ \frac{S^{1.03}}{\ln S}

. Wu and Yang [27] independently applied the idea of best polynomial approximation to entropy estimation, and obtained its minimax L 2 rate. The minimax lower bound part of Theorem 1 follows from Wu and Yang [27].…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

“…Wu and Yang [27] independently applied the idea of best polynomial approximation to entropy estimation, and obtained its minimax L 2 rate. The minimax lower bound part of Theorem 1 follows from Wu and Yang [27]. We also remark that, unlike the estimator we propose, the estimator in Wu and Yang [27] relies on knowledge of the support size S , which generally may not be known.…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Minimax Estimation of Functionals of Discrete Distributions

Jiao

Venkat

Han

et al. 2015

IEEE Trans. Inform. Theory

132

View full text Add to dashboard Cite

Abstract-We propose a general methodology for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions, where the support size S is unknown and may be comparable with or even much larger than the number of observations n. We treat the respective regions where the functional is "nonsmooth" and "smooth" separately. In the "nonsmooth" regime, we apply an unbiased estimator for the best polynomial approximation of the functional whereas, in the "smooth" regime, we apply a biascorrected version of the Maximum Likelihood Estimator (MLE).We illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures: the entropyWe obtain the minimax L2 rates for estimating these functionals. In particular, we demonstrate that our estimator achieves the optimal sample complexity n S/ ln S for entropy estimation. We also demonstrate that the sample complexity for estimating Fα(P ), 0 < α < 1, is n S 1/α / ln S, which can be achieved by our estimator but not the MLE. For 1 < α < 3/2, we show the minimax L2 rate for estimating Fα(P ) is (n ln n) −2(α−1) for infinite support size, while the maximum L2 rate for the MLE is n −2(α−1) . For all the above cases, the behavior of the minimax rate-optimal estimators with n samples is essentially that of the MLE (plug-in rule) with n ln n samples, which we term "effective sample size enlargement".We highlight the practical advantages of our schemes for the estimation of entropy and mutual information. We compare our performance with various existing approaches, and demonstrate that our approach reduces running time and boosts the accuracy. Moreover, we show that the minimax rate-optimal mutual information estimator yielded by our framework leads to significant performance boosts over the Chow-Liu algorithm in learning graphical models. The wide use of information measure estimation suggests that the insights and estimators obtained in this work could be broadly applicable.

show abstract

“…Another estimator proposed by Valiant and Valiant [26] has only been shown to achieve the minimax rate in the restrictive regime of

\frac{S}{\ln S} ≲ n ≲ \frac{S^{1.03}}{\ln S}

Section: Introduction and Main Resultsmentioning

confidence: 99%

Section: Introduction and Main Resultsmentioning

confidence: 99%

Minimax Estimation of Functionals of Discrete Distributions

Jiao

Venkat

Han

et al. 2015

IEEE Trans. Inform. Theory

132

View full text Add to dashboard Cite

show abstract

“…There exist extensive literature on this subject, and we refer to [22] for a detailed review, as well as the theory and Matlab/Python implementations of entropy and mutual information estimators that achieve the minimax rates in all the regimes of sample size and support size pairs. For the recent growing literature on information measure estimation in the high-dimensional regime, we refer to [23], [24], [25], [22], [26], [27], [28], [29].…”

Section: Problem Formulation and Main Resultsmentioning

confidence: 99%

Justification of Logarithmic Loss via the Benefit of Side Information

Jiao

Courtade

Venkat

et al. 2015

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-We consider a natural measure of relevance: the reduction in optimal prediction risk in the presence of side information. For any given loss function, this relevance measure captures the benefit of side information for performing inference on a random variable under this loss function. When such a measure satisfies a natural data processing property, and the random variable of interest has alphabet size greater than two, we show that it is uniquely characterized by the mutual information, and the corresponding loss function coincides with logarithmic loss. In doing so, our work provides a new characterization of mutual information, and justifies its use as a measure of relevance. When the alphabet is binary, we characterize the only admissible forms the measure of relevance can assume while obeying the specified data processing property. Our results naturally extend to measuring causal influence between stochastic processes, where we unify different causal-inference measures in the literature as instantiations of directed information.

show abstract

“…On the other hand, any estimator U E for U can be converted to a (not necessarily good) support size estimator by adding the number of observed symbols. Estimating the support size of an underlying distribution has been studied by both ecologists (1-3) and theoreticians (26)(27)(28); however, to make the problem nontrivial, all statistical models impose a lower bound on the minimum nonzero probability of each symbol, which is assumed to be known to the statistician. We discuss the connections and differences to our results in SI Appendix, section 5.…”

Section: Poissonmentioning

confidence: 99%

Optimal prediction of the number of unseen species

Orlitsky

Suresh

2016

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

131

View full text Add to dashboard Cite

Estimating the number of unseen species is an important problem in many scientific endeavors. Its most popular formulation, introduced by Fisher et al. [Fisher RA, Corbet AS, Williams CB (1943) J Animal Ecol 12(1):42−58], uses n samples to predict the number U of hitherto unseen species that would be observed if t · n new samples were collected. Of considerable interest is the largest ratio t between the number of new and existing samples for which U can be accurately predicted. In seminal works, Good and Toulmin [Good I, Toulmin G (1956) Biometrika 43(102):45−63] constructed an intriguing estimator that predicts U for all t ≤ 1. Subsequently, Efron and Thisted [Efron B, Thisted R (1976) Biometrika 63(3):435−447] proposed a modification that empirically predicts U even for some t > 1, but without provable guarantees. We derive a class of estimators that provably predict U all of the way up to t ∝ log n. We also show that this range is the best possible and that the estimator's mean-square error is near optimal for any t. Our approach yields a provable guarantee for the Efron−Thisted estimator and, in addition, a variant with stronger theoretical and experimental performance than existing methodologies on a variety of synthetic and real datasets. The estimators are simple, linear, computationally efficient, and scalable to massive datasets. Their performance guarantees hold uniformly for all distributions, and apply to all four standard sampling models commonly used across various scientific disciplines: multinomial, Poisson, hypergeometric, and Bernoulli product.species estimation | extrapolation model | nonparametric statistics S pecies estimation is an important problem in numerous scientific disciplines. Initially used to estimate ecological diversity (1-4), it was subsequently applied to assess vocabulary size (5, 6), database attribute variation (7), and password innovation (8). Recently, it has found a number of bioscience applications, including estimation of bacterial and microbial diversity (9-12), immune receptor diversity (13), complexity of genomic sequencing (14), and unseen genetic variations (15).All approaches to the problem incorporate a statistical model, with the most popular being the "extrapolation model" introduced by Fisher, Corbet, and Williams (16) in 1943. It assumes that n independent samples X n ≜ X 1 , . . . , X n were collected from an unknown distribution p, and calls for estimatingthe number of hitherto unseen symbols that would be observed if m additional samples X n+m n + 1 ≜ X n+1 , . . . , X n+m were collected from the same distribution.In 1956, Good and Toulmin (17) predicted U by a fascinating estimator that has since intrigued statisticians and a broad range of scientists alike (18). For example, in the Stanford University Statistics Department brochure (19), published in the early 1990s and slightly abbreviated here, Bradley Efron credited the problem and its elegant solution with kindling his interest in statistics. As we shall soon see, Efron, along with Ronald Thisted, ...

show abstract

Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation

Cited by 166 publications

References 58 publications

Minimax Estimation of Functionals of Discrete Distributions

Minimax Estimation of Functionals of Discrete Distributions

Justification of Logarithmic Loss via the Benefit of Side Information

Optimal prediction of the number of unseen species

Contact Info

Product

Resources

About