Probability Estimation in the Rare-Events Regime

Abstract-How can we effectively model situations with large alphabets? On a pragmatic level, any engineered system, be it for inference, communication, or encryption, requires working with a finite number of symbols. Therefore, the most straightforward model is a finite alphabet. However, to emphasize the disproportionate size of the alphabet, one may want to compare its finite size with the length of data at hand. More generally, this gives rise to scaling models that strive to capture regimes of operation where one anticipates such imbalance. Large alphabets may also be idealized as infinite. The caveat then is that such generality strips away many of the convenient machinery of finite settings. However, some of it may be salvaged by refocusing the tasks of interest, such as by moving from sequence to pattern compression, or by minimally restricting the classes of infinite models, such as via tail properties. In this paper we present an overview of models for large alphabets, some recent results, and possible directions in this area.

show abstract

“…In the rare events regime, WVK show in [3] that both M n,r and G n,r satisfy the same Poisson limit, in a strong sense:…”

Section: B Rare Probability Estimationmentioning

confidence: 98%

“…This is done notably by Wagner, Viswanath, and Kulkarni, whom we refer to as WVK henceforth, in [3]. Let P n be the law of np n (X n,1 ), i.e.…”

Section: A Rare-events Sourcesmentioning

confidence: 99%

Large alphabets: Finite, infinite, and scaling models

Ohannessian

Dahleh

2012

2012 46th Annual Conference on Information Sciences and Systems (CISS)

View full text Add to dashboard Cite

show abstract

“…In the above definition, the constraints in (15) and (16) ensure that v VV ′ is a (potentially improper) distribution histogram, whereas the constraints in (17) are based on computations what the statistics of the profile ϕ ϕ(µ) of the multiplicity vector µ µ(ψ) of a pattern ψ generated by a source with distribution histogram v ′ looks like. The size and the elements of the set Q are chosen such that a feasible vector v ′ can be found efficiently, yet so that one can guarantee nice properties of the solution.…”

Section: Definition 19 (Simplified Vv Estimate)mentioning

confidence: 99%

The Bethe and Sinkhorn approximations of the pattern maximum likelihood estimate and their connections to the Valiant-Valiant estimate

Vontobel

2014

2014 Information Theory and Applications Workshop (ITA)

View full text Add to dashboard Cite

Abstract-For estimating a source's distribution histogram, Orlitsky and co-workers have proposed the pattern maximum likelihood (PML) estimate, which says that one should choose the distribution histogram that has the largest likelihood of producing the pattern of the observed symbol sequence. It can be shown that finding the PML estimate is equivalent to finding the distribution histogram that maximizes the permanent of a certain non-negative matrix.However, in general this optimization problem appears to be intractable and so one has to compute suitable approximations of the PML estimate. In this paper, we discuss various efficient PML estimate approximation algorithms, along with their connections to the Valiant-Valiant estimate of the distribution histogram. These connections are established by associating an approximately doubly stochastic matrix with the Valiant-Valiant estimate and comparing this approximately doubly stochastic matrix with the doubly stochastic matrices that appear in the free energy descriptions of the PML estimate and its approximations.

show abstract

“…The composite-versus-composite case with growing alphabets is addressed in limited form by Wagner et al [17], who develop a probability estimator for the "rare-events" regime where underlying probabilities are all order Θ(n −1 ) and therefore alphabet size is order Θ(n). Other practical approaches may also be taken, see for example Orlitsky-Santhanam-Zhang (OSZ) [18], [19], support vector machines [20], and techniques from pattern recognition and machine learning [21].…”

Section: B Related Workmentioning

confidence: 99%

“…Let {p n , q n } be a sequence of pairs of distributions and denote by µ 2 n (x, y) the shadow (see [17]), i.e. distribution of the random vector np n (X n ),…”

Section: Appendix B Proofs: Section IVmentioning

confidence: 99%

Classification of Homogeneous Data With Large Alphabets

Kelly

Wagner

Tularak

et al. 2013

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

Given training sequences generated by two distinct, but unknown, distributions sharing a common alphabet, we study the problem of determining whether a third test sequence was generated according to the first or second distribution using only the training data. To better model sources such as natural language, for which the underlying distributions are difficult to learn, we allow the alphabet size to grow and therefore the probability distributions to change with the blocklength. Our primary focus is the situation in which the underlying probabilities are all of the same order, and in this regime we give conditions on the alphabet growth rate and distributions guaranteeing the existence of universally consistent tests, i.e. tests having a probability of error tending to zero with the blocklength for any underlying distributions. We show that some commonly used statistical tests are universally consistent provided the alphabet is sub-linear but these tests are inconsistent for linear growth rates. We then propose a classifier that is universally consistent with up-to quadratic alphabet growth and that no classifier can handle the case in which the alphabet grows quadratically or faster. If the tester is given the underlying distributions in place of the training data, we prove that consistent testing is possible regardless of the growth of the underlying alphabet. Our results are then used to illuminate the problem of classifying arbitrary (i.e. non-homogeneous) distributions on growing alphabets.

show abstract

Probability Estimation in the Rare-Events Regime

Cited by 27 publications

References 39 publications

Large alphabets: Finite, infinite, and scaling models

Large alphabets: Finite, infinite, and scaling models

The Bethe and Sinkhorn approximations of the pattern maximum likelihood estimate and their connections to the Valiant-Valiant estimate

Classification of Homogeneous Data With Large Alphabets

Contact Info

Product

Resources

About