Adaptive Relevance Matrices in Learning Vector Quantization

Abstract

We propose a new matrix learning scheme to extend relevance learning vector quantization (RLVQ), an efficient prototype-based classification algorithm, towards a general adaptive metric. By introducing a full matrix of relevance factors in the distance measure, correlations between different features and their importance for the classification scheme can be taken into account and automated, general metric adaptation takes place during training. In comparison to the weighted Euclidean metric used in RLVQ and its variations, a full matrix is more powerful to represent the internal structure of the data appropriately. Large margin generalization bounds can be transfered to this case leading to bounds which are independent of the input dimensionality. This also holds for local metrics attached to each prototype which corresponds to piecewise quadratic decision boundaries. The algorithm is tested in comparison to alternative LVQ schemes using an artificial data set, a benchmark multi-class problem from the UCI repository, and a problem from bioinformatics, the recognition of splice sites for C. elegans.

Ordinal regression based on learning vector quantization

“…Recently, Fouad and Tiňo (2012) extended generalized matrix learning vector quantization approach (Schneider, Biehl, & Hammer, 2009) to ordinal regression, where the order information among different categories is utilized in the selection of the prototypes to be adapted, as well as in updating of the selected prototypes. Given an (input, target class) training example, generalized matrix learning vector quantization identifies two prototypesthe closest prototype of the target class and the closest prototype among the prototypes with a different label.…”

“…While prototype pairing, given an input vector, of correct and incorrect prototypes naturally extends the generalized matrix learning vector quantization approach of Schneider et al (2009), the price to be paid is the need to brake the global linear class order into ordered pairs that are treated independently of each other. This is unnatural and the updating rules of the prototypes derived from this cost function cannot consistently guarantee the ordering relation among the prototypes.…”

“…In this way important dependencies between different input features as well as their individual importance for the classification task can be revealed and utilized (Schneider et al, 2009). In this approach, termed generalized matrix learning vector quantization (GMLVQ), again at each training step a single pair of prototypes is updated (the closest correct and the closest incorrect prototype to the current training input).…”

“…The learning rates are monotonically decreasing with time (Fouad & Tiňo, 2012;Schneider et al, 2009):…”

“…Commonly the magnitude of η is set smaller than that of ϵ, in order to achieve a slower time-scale of metric learning compared to the weight updates (Schneider et al, 2009). …”