Multi-objective genetic programming for manifold learning: balancing quality and dimensionality

Abstract

Manifold learning techniques have become increasingly valuable as data continues to grow in size. By discovering a lower-dimensional representation (embedding) of the structure of a dataset, manifold learning algorithms can substantially reduce the dimensionality of a dataset while preserving as much information as possible. However, state-of-the-art manifold learning algorithms are opaque in how they perform this transformation. Understanding the way in which the embedding relates to the original high-dimensional space is critical in exploratory data analysis. We previously proposed a Genetic Programming method that performed manifold learning by evolving mappings that are transparent and interpretable. This method required the dimensionality of the embedding to be known a priori, which makes it hard to use when little is known about a dataset. In this paper, we substantially extend our previous work, by introducing a multi-objective approach that automatically balances the competing objectives of manifold quality and dimensionality. Our proposed approach is competitive with a range of baseline and state-of-the-art manifold learning methods, while also providing a range (front) of solutions that give different tradeoffs between quality and dimensionality. Furthermore, the learned models are shown to often be simple and efficient, utilising only a small number of features in an interpretable manner.

Transparent Dimension Reduction by Feature Construction with Genetic Algorithm

“…This genetic algorithm is based on the idea that neighbors in a new low-dimensional space should have a similar ordering to that in the original high-dimensional space. This algorithm called GP-MaL-MO (Genetic Programming for Manifold Learning using a Multi-objective Approach) [21] is an extension of the GP-MaL algorithm [23]. This algorithm uses a multi-criterial fitness function to build a Pareto front of solutions.…”

“…The most common case of solving a dimension reduction problem is to use classical approaches such as PCA when there are a lot of features and it is simply impossible to work with a such large number of features, or t-SNE [18] when it is necessary to make some intuitive conclusions about a data distribution by visualization of feature space. On the other hand, there are a few works about transparent dimension reduction [19], [20], [21], [22]. They use an evolutionary approach to reduce the dimension of data and get interpretable by human features.…”

A Genetic Programming Encoder for Increasing Autoencoder Interpretability

“…These are commonly used as activation functions in neural networks, adding the capacity for non-linear learning. Existing GP for NLDR work has also used these [12,14].…”

“…GP has inherent potential for interpretability, because it evolves solutions that combine userselected terminals and functions. GP has recently been demonstrated to be a capable NLDR technique which produces functional mappings [12,14]. Some of these approaches have used a multi-tree GP representation with a custom fitness function for evaluating embedding quality [12-14, 22, 24], and other work has looked into GP specifically for autoencoding [16,19].…”

Rademacher Complexity for Enhancing the Generalization of Genetic Programming for Symbolic Regression

“…Many EC based techniques have been proposed for multi-objective learning such as nondominated sorting genetic algorithm II (NSGAII) [19], the Strength Pareto Evolutionary Algorithm 2 (SPEA2) [20] and the Multiobjective Evolutionary Algorithm Based on Decomposition (MOEA/D) [21]. The use of these multi-objective techniques has been investigated in variety of GP methods [22], [23], [24], [25], [26]. Bleuler et al [22] proposed a multi-objective GP method using SPEA2 to tackle the bloat issue in GP where the program/solution size and the training accuracy were considered as two independent objectives.…”