Uncertainty handling CMA-ES for reinforcement learning

Abstract

The covariance matrix adaptation evolution strategy (CMA-ES) has proven to be a powerful method for reinforcement learning (RL). Recently, the CMA-ES has been augmented with an adaptive uncertainty handling mechanism. Because uncertainty is a typical property of RL problems this new algorithm, termed UH-CMA-ES, is promising for RL. The UH-CMA-ES dynamically adjusts the number of episodes considered in each evaluation of a policy. It controls the signal to noise ratio such that it is just high enough for a sufficiently good ranking of candidate policies, which in turn allows the evolutionary learning to find better solutions. This significantly increases the learning speed as well as the robustness without impairing the quality of the final solutions. We evaluate the UH-CMA-ES on fully and partially observable Markov decision processes with random start states and noisy observations. A canonical natural policy gradient method and random search serve as a baseline for comparison.

Noisy optimization complexity under locality assumption

“…It considers a different definition of convergence (more on this later).R-EDA was proposed as a typical noisy optimization algorithm, easy to analyze and making a good approximation of real-world approaches [17]. The noisy optimization framework is described in Algorithm 1 and R-EDA is defined in Algorithm 2.…”

A New Multi-Agent Reinforcement Learning Method Based on Evolving Dynamic Correlation Matrix

“…Evolutionary tuning of reinforcement learning parameters has also been done by several researchers [51]- [55]. Eriksson et al [53] used evolutionary algorithms to tune the meta-parameters in a single-agent reinforcement learning algorithm.…”