Universal hypothesis testing in the learning-limited regime

Abstract

Given training sequences generated by two distinct, but unknown distributions sharing a common alphabet, we seek a classifier that can correctly decide whether a third test sequence is generated by the first or second distribution using only the training data. To model 'limited learning' we allow the alphabet size to grow and therefore probability distributions to change with the blocklength. We prove that a natural choice, namely a generalized likelihood ratio test, is universally consistent (has a probability of error tending to zero with the blocklength for all underlying distributions) when the alphabet size is sub-linear in the blocklength, but inconsistent for linear alphabet growth. For up-to quadratic alphabet growth, in a regime where all probabilities are of the same order, we prove the universally consistency of a new test and show there are no such tests when the alphabet grows quadratically or faster.

Classification with high-dimensional sparse samples

“…This classifier was shown in [1] to be asymptotically consistent when N = n and m = o(N 2 ). We now show, however, this classifier has zero generalized error exponent:…”

“…where η is a large positive constant. The definition of Π m is essentially the same as the α-large-alphabet source defined in [1], except that we allow the number of training and test samples to be different. While this assumption that all words are rare does not hold for English vocabulary, the insights and classifiers obtained for rare words will be used to improve the algorithms for the case when there are both frequent and rare words.…”

“…1) The numbers of training and test samples N, n have different effects on the performance, made precise in Theorem IV.1 and Theorem IV.2. 2) The ℓ 2 -norm based classifier investigated in [1], which compares the ℓ 2 distances from the empirical distribution of the test sequence to those of the two training sequences, is sub-optimal in that it has zero generalized error exponent, while a weighted coincidencebased classifier proposed in this paper has a non-zero generalized error exponent.…”

“…A widely-used performance criterion is asymptotic consistency: Given some dependence of N, n on m, does the probability of error decay to zero as m increases to infinity? A fundamental result with respect to this criterion was established in [1]: Assuming that the distribution on all symbols in the alphabet is of order 1/m, there exists an asymptotic consistent classifier if and only if m = o(n 2 ). Note that the result is established only for the case N = n.…”

Feature selection for composite hypothesis testing with small samples: Fundamental limits and algorithms

“…The high-dimensional model considered in this paper is similar to those investigated in [6,7] and the converse result in this paper is based on a similar proof technique.…”