On explaining machine learning models by evolving crucial and compact features

Abstract

Feature construction can substantially improve the accuracy of Machine Learning (ML) algorithms. Genetic Programming (GP) has been proven to be effective at this task by evolving non-linear combinations of input features. GP additionally has the potential to improve ML explainability since explicit expressions are evolved. Yet, in most GP works the complexity of evolved features is not explicitly bound or minimized though this is arguably key for explainability. In this article, we assess to what extent GP still performs favorably at feature construction when constructing features that are (1) Of small-enough number, to enable visualization of the behavior of the ML model; (2) Of small-enough size, to enable interpretability of the features themselves; (3) Of sufficient informative power, to retain or even improve the performance of the ML algorithm. We consider a simple feature construction scheme using three different GP algorithms, as well as random search, to evolve features for four ML algorithms, including support vector machines and random forest. Our results on 20 datasets pertaining to classification and regression problems show that constructing only two compact features can be sufficient to rival the use of the entire original feature set. We further find that a modern GP algorithm, GP-GOMEA, performs best overall. These results, combined with examples that we provide of readable constructed features and of 2D visualizations of ML behavior, lead us to positively conclude that GP-based feature construction still works well when explicitly searching for compact features, making it extremely helpful to explain ML models.

Learning a Formula of Interpretability to Learn Interpretable Formulas

“…The authors of [43] study whether modern model-based GP can be useful when particularly compact symbolic regression solutions are sought, to allow interpretability. A very different take to enable or improve interpretability is taken in [22,41,45], where interpretability is sought by means of feature construction and dimensionality reduction. In [22] in particular, MOGP is used, with solution size as a simple PHI.…”

Feature extraction by grammatical evolution for one-class time series classification

“…The majority of prior research on evolutionary techniques for TSC has applied to ECG time series [18], and sensor time series for fault detection [31]. Some studies, discussed below, propose general purpose feature extraction algorithms [15,23,52].…”

“…Finally, Virgolin et al [52] propose a GP-based algorithm for sequential feature construction but not on time series. While feature extraction concerns the extraction of features from raw data, feature construction concerns the transformation of existing features.…”

“…Several researchers have shown that GP can be used to enable interpretability of so-called black-box models (models that are not easy to interpret) [52]. Others have shown that evolutionary algorithms can be as effective as black-box models with the advantage of being more interpretable [29].…”

Towards crop yield estimation at a finer spatial resolution using machine learning methods over agricultural regions

“…Feature selection is one of the tasks used to eliminate redundant features and simplify calculations in machine learning (Gopika and Kowshalaya 2018;Virgolin et al 2020).…”