A Practical Method Based on Bayes Boundary-Ness for Optimal Classifier Parameter Status Selection

Abstract

We propose a novel practical method for finding the optimal classifier parameter status corresponding to the Bayes error (minimum classification error probability) through the evaluation of estimated class boundaries from the perspective of Bayes boundary-ness. While traditional methods approach classifier optimality from the angle of minimization of the estimated classification error probabilities, we approach it from the angle of optimality of the estimated classification boundaries. The optimal classification boundary consists solely of uncertain samples, whose posterior probability is equal for the two classes separated by the boundary. We refer to this essential characteristic of the boundary as "Bayes boundaryness", and use it to measure how optimal the estimated boundary is. Our proposed method achieves the optimal parameter status using the training data only once, in contrast to such traditional methods as Cross-Validation (CV), which demand separate validation data and often require a number of repetitions of training and validation. Moreover, it can be directly applied to any type of classifier, and potentially to any type of sample. In this paper, we first elaborate on our proposed method that implements the Bayes boundary-ness with an entropy-based uncertainty measure. Next, we analyze the mathematical characteristics of the uncertainty measure adopted. Finally, we evaluate the method through a systematic experimental comparison with CV-based Bayes boundary estimation, which is known to be highly reliable in the Bayes error estimation. From the analysis, we rigorously show the theoretical validity of our adopted uncertainty measure. Moreover, from the experiment, we successfully demonstrate that our method can closely approximate the CV-based Bayes boundary estimate and its corresponding classifier parameter status with only a single-shot training over the data in hand.

An Improved Boundary Uncertainty-Based Estimation for Classifier Evaluation

“…The empirical formalization of boundary uncertainty used in Proposal 1 actually left some unclear items, such as the possible confusion of whether to use {i * (x), j * (x)} (top given class indexes) or {i(x; Λ), j (x; Λ)} (top predicted class indexes). Such confusion was especially possible in the several intricate branching treatments that appeared in Proposal 1, as well as in the treatment of multiclass data [18]. In contrast, this paper introduces a more complete formalization of boundary uncertainty (Section 4.3).…”

“…Then, given a classifier status Λ, Proposal 1 empirically defined the following classifier evaluation metric that we estimated from the training set [18] and termed "boundary uncertainty": and achieving the minimum error probability (Bayes error). We describe this matter in more detail in later sections.…”

“…On the one hand, previous experiments showed that the restriction to N B (Λ) of target volumes used for kNN regression efficiently increases the accuracy ofÛ (1) (Λ) [18]. On the other hand, a discrete selection of "what is close to B(Λ) or not" unavoidably leads to the following dilemma: either taking too few samples that are not representative of B(Λ) or taking too many samples that are farther from B(Λ).…”

“…The limitations described above come from the intrinsic difficulty of estimating the error probability. To circumvent these limitations, we proposed finding the optimally trained classifier through a new classifier evaluation metric that is uniquely easy to estimate in principle, which we termed "boundary uncertainty" or alternately "Bayes-boundaryness" [16][17][18]. We chose the name "boundary uncertainty" because this evaluation metric measures the generalization ability of a classifier based on how equal class posterior probabilities are along the classifier boundary.…”

“…In addition to defining boundary uncertainty, our previous work also proposed a boundary uncertainty estimation method that here we refer to as "Proposal 1." Through experiments and theoretical analysis, we showed that Proposal 1 could perform classifier evaluation using neither sample distribution assumptions nor data resampling methods [16][17][18]. However, Proposal 1 relied on largely unclear settings and heavy treatment that seriously limited both its accuracy and its scalability.…”