Rejection Strategies for Learning Vector Quantization – A Comparison of Probabilistic and Deterministic Approaches

Abstract

Abstract. We present prototype-based classification schemes, e. g. learning vector quantization, with cost-function-based and geometrically motivated reject options. We evaluate the reject schemes in experiments on artificial and benchmark data sets. We demonstrate that reject options improve the accuracy of the models in most cases, and that the performance of the proposed schemes is comparable to the optimal reject option of the Bayes classifier in cases where the latter is available. MotivationPowerful machine learning methods such as recent learning vector quantization (LVQ) models based on cost functions or support vector machines and linear time approximations thereof provide state of the art classification algorithms for automated data analysis [1,2,3,4]. Their linear time complexity, high accuracy, and excellent generalization ability make them suitable also for large data sets. However, generalization bounds and training algorithms rely on the assumption of data being i.i.d. This limits the suitability for big data analysis, streaming data which displays a trend, in presence of outliers, or regions of strong overlap in the data. These cases require enhancing the classifier by measures of certainty that a model has taken a classification decision for a certain point or a data region. Such reject options constitute a first step towards incremental adaptation of the model complexity tailored to data regions with a high degree of uncertainty.While there exist popular extensions of SVM to provide a confidence value of the classification [5,6,7] and first models have been proposed for distance-based k-nearest neighbor approaches [8], only few approaches address prototype-based classifiers [9, 10] thereby lacking a comparison to theoretically motivated alternatives such as explicit stochastic models. In this contribution, we are interested in efficient, online-computable reject options for LVQ classifiers and their behavior in comparison to mathematically well founded statistical models. For this purpose, we address the cost-function based models generalized LVQ (GLVQ) [11] and generalized matrix LVQ (GMLVQ) [1] as well as the probabilistic counterpart robust soft LVQ (RSLVQ) [2]. We propose simple geometric reject options for these models which can be computed efficiently in online scenarios, and we compare these reject options to more costly alternatives based on a probabilistic modeling and an optimum reject option for the Bayes classifier [12].

Local Rejection Strategies for Learning Vector Quantization

“…Albeit there exist few approaches which model LVQ classifiers by means of class probabilities such as RSLVQ [15], it is unclear whether such discriminative models converge to the correct underlying class distributions, and most popular LVQ schemes are based on deterministic decision models only instead of a reference to class probabilities [5,12,17]. Recently [4,5] it has been analysed if alternative real-valued outputs correlated to the deterministic classification model can take the role of a certainty value for a reject option: examples include the distance of a data vector to the closest decision boundary, prototype. Interestingly, using simple thresholds, these measures offer classification schemes with a reject option with the quality close to optimum Bayesian decisions in simple model cases [4,5].…”

“…Recently [4,5] it has been analysed if alternative real-valued outputs correlated to the deterministic classification model can take the role of a certainty value for a reject option: examples include the distance of a data vector to the closest decision boundary, prototype. Interestingly, using simple thresholds, these measures offer classification schemes with a reject option with the quality close to optimum Bayesian decisions in simple model cases [4,5].…”

“…Here we present a greedy optimization method to adaptively determine local thresholds for an LVQ classifier based on given data. We compare the resulting local rejection strategies to the global counterparts as proposed in [4,5] using several benchmarks and one artificial data set. We show that local thresholds outperform their global counterpart, approximating the optimal reject option of Chow [3] in cases where the latter is available.…”

Efficient rejection strategies for prototype-based classification

“…Measures based on probabilities often either require a probabilistic classification model [5,36] or a probabilistic model on top of the trained classifier to estimate the probabilities [11,29]. Both approaches are computationally expensive.…”

Median Variants of Prototype Based Learning Vector Quantization: Methods for Classification of General Proximity Data

“…for several parameter choices γ 8 . For demonstration purposes we also consider the dissimilarity based on the Gaussian kernel d Gauss , for which we already know from section 3.2 that it is rank equivalent to the squared Euclidean distance.…”