Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization

Abstract

We present Reward-Switching Policy Optimization (RSPO), a paradigm to discover diverse strategies in complex RL environments by iteratively finding novel policies that are both locally optimal and sufficiently different from existing ones. To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic rewards via a trajectory-based novelty measurement during the optimization process. When a sampled trajectory is sufficiently distinct, RSPO performs standard policy optimization with extrinsic rewards. For trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward to promote exploration. Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent particle-world tasks and MuJoCo continuous control to multi-agent stag-hunt games and StarCraftII challenges.

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

“…SMERL (Kumar et al 2020): SMERL maximizes a weighted combination of intrinsic rewards and extrinsic rewards when the return of extrinsic reward is greater than some given threshold. RSPO (Zhou et al 2022): RSPO is an iterative algorithm for discovering a diverse set of quality strategies. It toggles between extrinsic and intrinsic rewards based on a trajectorybased novelty measurement.…”

“…However, their method operates in an unsupervised manner, without external rewards. More recently, RSPO (Zhou et al 2022) was proposed to derive diverse strategies. However, it requires multiple training stages, which results in poor sample efficiency -our method trains diverse strategies simultaneously which reduces sample complexity.…”

“…2. The task inherently allows multiple optimal solutions, such as a maze offering two equally efficient paths (Osa, Tangkaratt, and Sugiyama 2021;Zhou et al 2022).…”

“…Sample Efficiency: Unlike predecessors like RSPO (Zhou et al 2022) and RPG (Tang et al 2021), which necessitated multiple networks and training phases, DGPO employs a shared network for concurrent learning of strategies, resulting in superior sample efficiency.…”

Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation

“…Recent works have demonstrated that diversity-driven policies can extrapolate to new environments through the few-shot adaptation (Eysenbach et al 2018;Kumar et al 2020;Osa, Tangkaratt, and Sugiyama 2021;Parker-Holder et al 2020;Zhou et al 2022). While the policy population with different behavior characteristics can generalize to different environment variations, the learned policies may result in potential safety problems in practical scenarios like real-world systems, as the behaviors of the diverse policies are unpredictable.…”