Adaptive evolution strategy with ensemble of mutations for Reinforcement Learning

Ajani, Oladayo S.; Mallipeddi, Rammohan

doi:10.1016/j.knosys.2022.108624

Cited by 17 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ajani and Mallipeddi [111] discussed the advantages and disadvantages of using Cauchy or Gaussian distribution for mutation. The authors stated that the Cauchy distribution is preferred when the search is far away from the optimal solution, and Gaussian is better near the global optimum.…”

Section: B Explorationmentioning

confidence: 99%

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Majid

Saaybi

François-Lavet

et al. 2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) and evolution strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths and weaknesses of DRL versus ESs, an analysis of their respective capabilities and limitations is provided. After presenting their fundamental concepts and algorithms, a comparison is provided on key aspects, such as scalability, exploration, adaptation to dynamic environments, and multiagent learning. Current research challenges are also discussed, including sample efficiency, exploration versus exploitation, dealing with sparse rewards, and learning to plan. Then, the benefits of hybrid algorithms that combine DRL and ESs are highlighted.

show abstract

Section: B Explorationmentioning

confidence: 99%

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Majid

Saaybi

François-Lavet

et al. 2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Consequently, in comparison to backpropagationbased algorithms, ES algorithms often necessitate more rollouts and are susceptible to local optima [7]. Previous efforts to enhance the performance of ES algorithms have included adding regularization terms to encourage exploration [8] or employing various noise perturbations [1]. While ES algorithms are generally less sensitive to hyperparameter settings than traditional DRL [30], prior research has shown that hyperparameter fine-tuning can still be effective for them.…”

Section: Related Workmentioning

confidence: 99%

BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization

Wang,

Zhu,

Wang

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Evolutionary reinforcement learning (ERL) algorithms recently raise attention in tackling complex reinforcement learning (RL) problems due to high parallelism, while they are prone to insufficient exploration or model collapse without carefully tuning hyperparameters (aka meta-parameters). In the paper, we propose a general meta ERL framework via bilevel optimization (BiERL) to jointly update hyperparameters in parallel to training the ERL model within a single agent, which relieves the need for prior domain knowledge or costly optimization procedure before model deployment. We design an elegant meta-level architecture that embeds the inner-level’s evolving experience into an informative population representation and introduce a simple and feasible evaluation of the meta-level fitness function to facilitate learning efficiency. We perform extensive experiments in MuJoCo and Box2D tasks to verify that as a general framework, BiERL outperforms various baselines and consistently improves the learning performance for a diversity of ERL algorithms.

show abstract

“…Several bio-inspired robotic applications and learning tasks often require the agents to adapt to the uncertainties in their environment. In simulated environments, Deep Reinforcement Learning (DRL) has proven to be successful at learning tasks across a wide range of domains such as games [1], rehabilitation [2], locomotion [3], production optimization [4], control [5][6][7], etc. However, in real-world environments, the deployment of DRL is often limited due to changing environmental conditions, an intrinsic feature mostly associated with real-world applications.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Domain Randomization in Deep Reinforcement Learning Locomotion Tasks

Ajani,

Hur,

Mallipeddi

2023

Mathematics

Self Cite

View full text Add to dashboard Cite

Domain randomization in the context of Reinforcement learning (RL) involves training RL agents with randomized environmental properties or parameters to improve the generalization capabilities of the resulting agents. Although domain randomization has been favorably studied in the literature, it has been studied in terms of varying the operational characters of the associated systems or physical dynamics rather than their environmental characteristics. This is counter-intuitive as it is unrealistic to alter the mechanical dynamics of a system in operation. Furthermore, most works were based on cherry-picked environments within different classes of RL tasks. Therefore, in this work, we investigated domain randomization by varying only the properties or parameters of the environment rather than varying the mechanical dynamics of the featured systems. Furthermore, the analysis conducted was based on all six RL locomotion tasks. In terms of training the RL agents, we employed two proven RL algorithms (SAC and TD3) and evaluated the generalization capabilities of the resulting agents on several train–test scenarios that involve both in-distribution and out-distribution evaluations as well as scenarios applicable in the real world. The results demonstrate that, although domain randomization favors generalization, some tasks only require randomization from low-dimensional distributions while others require randomization from high-dimensional randomization. Hence the question of what level of randomization is optimal for any given task becomes very important.

show abstract

Adaptive evolution strategy with ensemble of mutations for Reinforcement Learning

Cited by 17 publications

References 18 publications

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization

Evaluating Domain Randomization in Deep Reinforcement Learning Locomotion Tasks

Contact Info

Product

Resources

About