Classification of Homogeneous Data With Large Alphabets

Abstract

Given training sequences generated by two distinct, but unknown, distributions sharing a common alphabet, we study the problem of determining whether a third test sequence was generated according to the first or second distribution using only the training data. To better model sources such as natural language, for which the underlying distributions are difficult to learn, we allow the alphabet size to grow and therefore the probability distributions to change with the blocklength. Our primary focus is the situation in which the underlying probabilities are all of the same order, and in this regime we give conditions on the alphabet growth rate and distributions guaranteeing the existence of universally consistent tests, i.e. tests having a probability of error tending to zero with the blocklength for any underlying distributions. We show that some commonly used statistical tests are universally consistent provided the alphabet is sub-linear but these tests are inconsistent for linear growth rates. We then propose a classifier that is universally consistent with up-to quadratic alphabet growth and that no classifier can handle the case in which the alphabet grows quadratically or faster. If the tester is given the underlying distributions in place of the training data, we prove that consistent testing is possible regardless of the growth of the underlying alphabet. Our results are then used to illuminate the problem of classifying arbitrary (i.e. non-homogeneous) distributions on growing alphabets.

Weak Convergence Analysis of Asymptotically Optimal Hypothesis Tests

“…These results do not provide general tools to approximate the false alarm probability in the finite sample setting except in the case of uniform null disribution. Numerous other similar examples of asymptotically optimal hypothesis tests are found in literature (see, e.g., [3], [7]- [13]). Nevertheless, in a practical experiment involving such hypothesis tests, one has access only to a finite number of observations, and the metric of practical interest is the actual error probability with a finite number of samples rather than the error exponent or asymptotic consistency.…”

Generalized Error Exponents for Small Sample Universal Hypothesis Testing

“…Substituting this into (39) leads to (40) When , we obtain Substituting this into (39) leads to (41) It follows from the bounds (40), (41) and that . Thus, the denominator of (39) satisfies Substituting this into (39) leads to Consequently, To obtain a refined approximation, let , which implies (42) An approximation for will be obtained: since , we have that the numerator and denominator in the summand of (39) satisfy Thus, Substituting this and (42) into (39) leads to which gives (43) The integration in (38) is now carried out along the closed contour given by :…”

“…To obtain tight bounds, we use a technique similar to the expurgating method in [40]. The distributions used in proving the bounds are constructed using the mixing of indistinguishable distributions method (see e.g., [5], [41]). …”

“…We use the mixing of indistinguishable distributions method previously used in proving hardness results for composite and hypothesis testing problems [3], [5], [41]. First, construct a collection of distributions so that for each distribution , the likelihood ratio has a simple expression.…”

“…For arbitrary nonuniform null distribution, one possible approach is to choose a different weight. As the results in [3], [41], [43], and [44] implies, the key is to analyze large probability and small probability symbols. A unified result on the large deviations for separable statistics for both large and small probability symbols would serve as a basis for choosing the weight.…”