Optimal “Anti-Bayesian” Parametric Pattern Classification Using Order Statistics Criteria

Abstract

Abstract. The gold standard for a classifier is the condition of optimality attained by the Bayesian classifier. Within a Bayesian paradigm, if we are allowed to compare the testing sample with only a single point in the feature space from each class, the optimal Bayesian strategy would be to achieve this based on the (Mahalanobis) distance from the corresponding means. The reader should observe that, in this context, the mean, in one sense, is the most central point in the respective distribution. In this paper, we shall show that we can obtain optimal results by operating in a diametrically opposite way, i.e., a so-called "anti-Bayesian" manner. Indeed, we shall show the completely counter-intuitive result that by working with a very few (sometimes as small as two) points distant from the mean, one can obtain remarkable classification accuracies. Further, if these points are determined by the Order Statistics of the distributions, the accuracy of our method, referred to as Classification by Moments of Order Statistics (CMOS), attains the optimal Bayes' bound! This claim, which is totally counter-intuitive, has been proven for many uni-dimensional, and some multi-dimensional distributions within the exponential family, and the theoretical results have been verified by rigorous experimental testing. Apart from the fact that these results are quite fascinating and pioneering in their own right, they also give a theoretical foundation for the families of Border Identification (BI) algorithms reported in the literature.

A Novel Border Identification Algorithm Based on an “Anti-Bayesian” Paradigm

“…In line with the newly proposed OS-based anti-Bayesian classifiers [5,6,7], we created the "border" set by selecting those patterns which are close to the true border of the alternate class. The classification is achieved with regard to these border patterns alone, and the size of this set is very small, in some cases, as small as five from each class.…”

“…The intriguing feature of these few points is that they lie close to the boundary and not to the mean, implying an "anti-Bayesian" philosophy [5,6,7].…”

On Achieving Near-Optimal “Anti-Bayesian” Order Statistics-Based Classification for Asymmetric Exponential Distributions

“…Earlier, in [1] and [2], we showed that we could obtain optimal results by an "anti-Bayesian" paradigm by using the OS. Interestingly enough, the novel methodology that we propose, referred to as Classification by Moments of Order Statistics (CMOS), is computationally not any more complex than working with the Bayesian paradigm itself.…”

“…The interesting point about these indicators is that some of them are quite unrelated to the traditional moments themselves, and in spite of this, have not been used in achieving PR. The amazing fact, demonstrated in [3] is that OS can be used in PR, and that such classifiers operate in a completely "anti-Bayesian" manner, i.e., by only considering certain outliers of the distribution.…”

“…This paper considers the use of Order Statistics (OS) in the theory of Pattern Recognition (PR). The pioneering work on using OS for classification was presented in [1] for the Uniform distribution, where it was shown that optimal PR can be achieved in a counter-intuitive manner, diametrically opposed to the Bayesian paradigm, i.e., by comparing the testing sample to a few samples distant from the mean -which is distinct from the optimal Bayesian paradigm. In [2], we showed that the results could be extended for a few symmetric distributions within the exponential family.…”