Distance Measures for Prototype Based Classification

Abstract

Abstract. The basic concepts of distance based classification are introduced in terms of clear-cut example systems. The classical k-NearestNeigbhor (kNN) classifier serves as the starting point of the discussion. Learning Vector Quantization (LVQ) is introduced, which represents the reference data by a few prototypes. This requires a data driven training process; examples of heuristic and cost function based prescriptions are presented. While the most popular measure of dissimilarity in this context is the Euclidean distance, this choice is frequently made without justification. Alternative distances can yield better performance in practical problems. Several examples are discussed, including more general Minkowski metrics and statistical divergences for the comparison of, e.g., histogram data. Furthermore, the framework of relevance learning in LVQ is presented. There, parameters of adaptive distance measures are optimized in the training phase. A practical application of Matrix Relevance LVQ in the context of tumor classification illustrates the approach.

Combining offline and online classifiers for life-long learning

“…LVQ can be extended by the powerful concept of metric learning, which got popular in distance-based classification lately [29], [30], [31]. Especially, there exists a generalisation of the GLVQ towards a general quadratic form (x−w j ) T Λ(x− w j ) with positive semi-definite matrix Λ which is proposed under the acronym GMLVQ [27].…”

“…We will rely on powerful formalisations of LVQ learning in terms of cost functions, such as proposed in the approaches [25], [26], [27], and we will use its recent extensions to incremental learning like the incremental, online LVQ (ioLVQ) [28]. Notably, state of the art LVQ classifiers usually integrate the powerful concept of metric learning [29], [30], [31]. For a comparison we refer to state of the art incremental learners, such as incremental support vector machines (iSVM)…”

Aspects in Classification Learning - Review of Recent Developments in Learning Vector Quantization

“…Several attempts were proposed to make progress regarding this problem ranging from intelligent initialization to the harmonic to minimum LVQ algorithm (H2M-LVQ, [71]). This latter approach starts with a different cost function compared to GLVQ incorporating the harmonic average distance instead of d + and d − in (9). According to this average, the whole distance information between the presented data sample and all prototypes is taken into account, which reduces the initialization sensitivity.…”

“…Unfortunately, original GLVQ as proposed in [77] does not optimize the classification error rather than the cost function E GLV Q (W, V ) from (9). Hence, the performance cannot judged consistently neither in terms of the statistical quantities provided by the confusion matrix nor by the ROC analysis.…”

“…Drawbacks of this approach are the sensitivity with respect to outliers and the resulting risk of overfitting and the computational effort in the working phase. There exist several approaches to reduce these problems using condensed training sets and improved selection strategies [18,39,110] as pointed out in [9]. Nevertheless, kNN frequently serves as a baseline.…”