An Introduction to Feature Extraction

Abstract

This chapter introduces the reader to the various aspects of feature extraction covered in this book. Section 1 reviews definitions and notations and proposes a unified view of the feature extraction problem. Section 2 is an overview of the methods and results presented in the book, emphasizing novel contributions. Section 3 provides the reader with an entry point in the field of feature extraction by showing small revealing examples and describing simple but effective algorithms. Finally, Section 4 introduces a more theoretical formalism and points to directions of research and open problems. 3 The outcome, also called target value, may be binary for a 2-class classification problem, categorical for a multi-class problem, ordinal or continuous for regression. 4 It is sometimes necessary to make the distinction between "raw" input variables and "features" that are variables constructed for the original input variables. We will make it clear when this distinction is necessary. Discussion. Many definitions of relevance have been provided in the literature. Kohavi and John (Kohavi and John, 1997) make a distinction between strongly and weakly relevant features. We recall below those definitions:

Frontal alpha oscillations distinguish leaders from followers: Multivariate decoding of mutually interacting brains

“…general linear model with a binomial cost and logit link function) (Stauffer, 2008), the two classes, 1) interactive and 2) computer control, were predicted based on the forward sequentially selected features (Guyon and Elisseeff, 2003). The classifier thus selected a subset of features from the data matrix that best predicted the class they belonged to, by sequentially selecting features until there was no improvement in prediction.…”

A systematic review of predictive modeling for bronchiolitis

“…This approach may miss an opportunity to identify combinations of predictors that together have good predictive power, when each individual predictor has little or no predictive power by itself [263]. This is particularly the case if non-linear interactions exist among multiple predictors.…”

Approaches to working in high-dimensional data spaces: gene expression microarrays

“…The vast number of feature subsets necessitates applying heuristic search techniques, with various accuracy/computation tradeoffs (Guyon and Elisseeff, 2003). Filtering methods apply knowledge of the class labels to evaluate the discrimination power either of individual genes (univariate) or collections of genes (multivariate), based on criteria such as signal-to-noise ratio, correlation measures, and mutual information, before classifier training.…”

“…These methods can improve predictive power by capturing higher order (and complex, nonlinear) joint feature effects. Perhaps the simplest example is the 'noisy XOR problem', for which two individual features and their linear combinations have no discrimination power, but a simple nonlinear combination is perfectly discriminating (Duda et al, 2001;Guyon and Elisseeff, 2003; Figure 2). …”

“…Other algorithms apply a backward search, which starts from the full space and then eliminates features. Floating (bidirectional) searches, which combine forward and backward steps, and more complex simulated annealing and genetic algorithms, can also be applied (Guyon and Elisseeff, 2003). Finally, there are methods that integrate classifier training and feature selection, such as decision trees, which essentially perform forward feature selection while growing a tree and backward elimination while pruning the tree (Duda et al, 2001).…”