A scalable species-based genetic algorithm for reinforcement learning problems

Abstract

Reinforcement Learning (RL) methods often rely on gradient estimates to learn an optimal policy for control problems. These expensive computations result in long training times, a poor rate of convergence, and sample inefficiency when applied to real-world problems with a large state and action space. Evolutionary Computation (EC)-based techniques offer a gradient-free apparatus to train a deep neural network for RL problems. In this work, we leverage the benefits of EC and propose a novel variant of genetic algorithm called SP-GA which utilizes a species-inspired weight initialization strategy and trains a population of deep neural networks, each estimating the Q-function for the RL problem. Efficient encoding of a neural network that utilizes less memory is also proposed which provides an intuitive mechanism to apply Gaussian mutations and single-point crossover. The results on Atari 2600 games outline comparable performance with gradient-based algorithms like Deep Q-Network (DQN), Asynchronous Advantage Actor Critic (A3C), and gradient-free algorithms like Evolution Strategy (ES) and simple Genetic Algorithm (GA) while requiring far fewer hyperparameters to train. The algorithm also improved certain Key Performance Indicators (KPIs) when applied to a Remote Electrical Tilt (RET) optimization task in the telecommunication domain.

Special issue on evolutionary machine learning

“…The second paper A scalable species-based genetic algorithm for reinforcement learning problems by Seth et al . (2022) proposes a novel genetic algorithm (GA) variant called species-based GA (SP-GA) which utilizes a species-inspired weight initialization strategy and trains a population of deep neural networks, each estimating the Q-function for the RL problem. The authors’ results on Atari 2600 games demonstrate that the performance of SP-GA is comparable with gradient-based algorithms like deep Q-network, asynchronous advantage actor critic and gradient-free algorithms like evolution strategy (ES) and simple GA while requiring far fewer hyperparameters to train.…”

“…The second paper A scalable species-based genetic algorithm for reinforcement learning problems by Seth et al (2022) proposes a novel genetic algorithm (GA) variant called species-based GA (SP-GA) which utilizes a species-inspired weight initialization strategy and trains a population of deep neural Cite this article: K. Mason and P. Mannion. Special issue on evolutionary machine learning.…”