Information Theoretical Kernels for Generative Embeddings Based on Hidden Markov Models

Abstract

Abstract. Many approaches to learning classifiers for structured objects (e.g., shapes) use generative models in a Bayesian framework. However, state-of-the-art classifiers for vectorial data (e.g., support vector machines) are learned discriminatively. A generative embedding is a mapping from the object space into a fixed dimensional feature space, induced by a generative model which is usually learned from data. The fixed dimensionality of these feature spaces permits the use of state of the art discriminative machines based on vectorial representations, thus bringing together the best of the discriminative and generative paradigms.Using a generative embedding involves two steps: (i) defining and learning the generative model used to build the embedding; (ii) discriminatively learning a (maybe kernel) classifier on the adopted feature space. The literature on generative embeddings is essentially focused on step (i), usually adopting some standard off-the-shelf tool (e.g., an SVM with a linear or RBF kernel) for step (ii). In this paper, we follow a different route, by combining several Hidden Markov Models-based generative embeddings (including the classical Fisher score) with the recently proposed non-extensive information theoretic kernels. We test this methodology on a 2D shape recognition task, showing that the proposed method is competitive with the state-of-art.

Dissecting psychiatric spectrum disorders by generative embedding

“…This problem of feature selection poses a key research theme in machine learning and requires appropriate techniques for dimensionality reduction. One attractive option is to embed the data in a feature space that is constructed using a generative model ( Lasserre et al, 2006; Martins et al, 2010; Minka, 2005; Perina et al, 2010 ). This feature space, referred to as a generative score space , embodies a model-guided dimensionality reduction of the observed data.…”

Generative Embedding for Model-Based Classification of fMRI Data

“…Generative kernels have been fruitfully exploited in a range of applications [54]–[66] and define an active area of research [67]–[70]. In the special case of generative embedding, a generative kernel is used to construct a generative score space .…”

“…Generative embedding represents a special case of using generative kernels for classification, such as the P -kernel [52] or the Fisher kernel [53] . Generative kernels have been fruitfully exploited in a range of applications [54] – [66] and define an active area of research [67] – [70] . In the special case of generative embedding, a generative kernel is used to construct a generative score space .…”

Renal Cancer Cell Classification Using Generative Embeddings and Information Theoretic Kernels

“…Instead of relying on standard kernels, we investigate the use of the recently introduced information theoretic (IT) kernels [15] as a similarity measure between objects in the generative embedding space. The main idea is that, with such kernels, we can exploit the probabilistic nature of the generative embeddings, improving even more the classification results of the hybrid approaches -this has been already shown in other classification contexts [3,16].…”