Distribution estimation consistent in total variation and in two types of information divergence

Abstract

The problem of the nonparametric estimation of a probability distribution is considered from three viewpoints: the consistency in total variation, the consistency in information divergence, and consistency in reversed order information divergence. These types of consistencies are relatively strong criteria of convergence, and a probability distribution cannot he consistently estimated in either type of convergence without any restrictions on the class of probability distributions allowed. Histogram-based estimators of distribution are presented which, under certain conditions, converge in total variation, in information divergence, and in reversed order information divergence to the unknown probability distribution. Some a priori information about the true probability distribution is assumed in each case. As the concept of consistency in information divergence is stronger than that of convergence in total variation, additional assumptions are imposed in the cases of informational divergences.

Estimation of the information by an adaptive partitioning of the observation space

“…1.5]). A step toward balancing the influence of all partition cells was undertaken by Barron et al [1] who proposed nonuniform and possibly nonproduct partitions into cells of a constant dominating probability rather than of a constant Lebesgue volume. An even more inspiring method (applicable, however, only to one-dimensional spaces) is the finite partitioning by sample quantiles, see [3]; the sample 1=n-quantiles are equivalent to the spacings discussed in [2], see also [16].…”

“…Step 0 Let R (1) n = P n 2 Q n , where P n and Q n are partitions of IR into r intervals specified by the marginal sample quantiles 1 j r 0 1: F n (x) = P n ((01; x) 2 IR); G n (x) = P n (IR 2 (01; x)) are the marginal distribution functions of P n : In other words, R (1) n is the product of statistically equivalent marginal blocks.…”

How to get Central Limit Theorems for global errors of estimates

“…While this method works for the general estimates defined by (3) it is limited to the integrated square error (p = 2)…”

Adaptive coding and prediction of sources with large and infinite alphabets

“…p(x 1 1 11x t ) r ;p (x 1 111x t ): (2) Related to this quantity we define the maximum average divergence (at time t) by …”

“…These results imply that, for infinite alphabet, without any condition on the source distribution it is impossible to have universal source code. On the other hand, there are some necessary and sufficient conditions for existence of universal codes (and distribution estimation) suggested by Kieffer [17] and Barron, Györfi and van der Meulen [2] (see also Kieffer and Yang [19] and He and Yang [14] who showed that any 0018-9448/$25.00 © 2008 IEEE collection satisfying Keiffer's condition from [17] can be compressed by grammar codes). In this correspondence, we show that the same, in spirit, results are valid for letterwise coding and prediction.…”