Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization

Abstract

As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss-function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.Preprint. Under review.

Optimizing loss functions through multi-variate taylor polynomial parameterization

“…Genetic Loss Optimization (GLO) [4] provided an initial study into metalearning of loss functions. GLO is based on a two-phase approach that (1) evolves a function structure using a tree representation, and (2) optimizes a structure's coefficients using an evolutionary strategy.…”

“…In doing so, it makes it possible to regularize the solutions automatically. Genetic Loss Optimization (GLO) [4] provided an initial implementation of this idea using a combination of genetic programming and evolutionary strategies.…”

Meta-learning PINN loss functions

“…In this regard, a recent research direction is concerned with loss function meta-learning, with diverse applications in supervised and reinforcement learning [18][19][20][21][22][23][24][25][26][27][28][29][30][31]. Although different works utilize different meta-learning techniques and have different goals, it has been shown that loss functions obtained via meta-learning can lead to an improved convergence of the gradient-descent-based optimization.…”

Searching for Robustness: Loss Learning for Noisy Classification Tasks

“…Meta-learning and Loss Learning Meta-learning, also known as learning to learn, has been applied for a wide variety of purposes as summarized in [12]. Of particular relevance is meta-learning of loss functions, which has been studied for various purposes including providing differentiable surrogates of non-differentiable objectives [14], optimizing efficiency and asymptotic performance of learning [17,2,13,39,7,8], and improving robustness to train/test domain-shift [1,24]. We are particularly interested in learning white-box losses for efficiency and improved task-transferability compared to neural network alternatives [2,13,1,24].…”

“…We are particularly interested in learning white-box losses for efficiency and improved task-transferability compared to neural network alternatives [2,13,1,24]. Meta-learning of white-box learner components has been demonstrated for optimizers [38], activation functions [28] and losses for accelerating conventional supervised learning [7,8]. We are the first to demonstrate the value of automatic loss function discovery for general purpose label-noise robust learning.…”