On the minimum probability of classification error through effective cardinality comparison

Abstract

Abstract-This work proposes a method for estimating a lower (Bayesian) bound for classification error rate in two-class problems. This lower bound is typically inferred through specific classifier structures. By contrast, the proposed approach is based on "collision" (quadratic) entropy estimator, deployed in the very pragmatic form of simple coincidence counters. To properly introduce this new approach, we first discuss the concept of sets' effective cardinality in view of basic concepts of probability and set theory. The usefulness as well as some limitations of the analysis based on effective cardinality are exemplified throughout the text.

On the Information Content of Predictions in Word Analogy Tests

“…These connections to the Shannon's seminal work are further discussed in the Appendix. Notice that, as in [18], accuracies can be alternatively thought in terms of entropy, through the concept of effective cardinality, as illustrated in the Appendix. Accordingly, any classification/detection problem, with a given accuracy, A, is analogous to another problem of finding a single target element in a chimeric set of C = 1/A equally likely ones, where C is the effective cardinality of the chimeric set, and H = log 2 (C) is the corresponding entropy, or its average information content, in bits.…”

“…Entropy measures reflect incertitudes about w b , and are theoretically related to accuracies [18]. Indeed, average accuracies can be expressed as…”

“…In this case, C 1 is not an actual set cardinality, but it plays the role of an effective cardinality that intuitively reflects the difficulty level of the guessing game, in this context. More generally, effective cardinality can be a useful dual for the entropy concept itself, as explained in [18] and references therein.…”