Reinforcement Learning Agents acquire Flocking and Symbiotic Behaviour in Simulated Ecosystems

Abstract

In nature, group behaviours such as flocking as well as crossspecies symbiotic partnerships are observed in vastly different forms and circumstances. We hypothesize that such strategies can arise in response to generic predator-prey pressures in a spatial environment with range-limited sensation and action. We evaluate whether these forms of coordination can emerge by independent multi-agent reinforcement learning in simple multiple-species ecosystems. In contrast to prior work, we avoid hand-crafted shaping rewards, specific actions, or dynamics that would directly encourage coordination across agents. Instead we test whether coordination emerges as a consequence of adaptation without encouraging these specific forms of coordination, which only has indirect benefit. Our simulated ecosystems consist of a generic food chain involving three trophic levels: apex predator, mid-level predator, and prey. We conduct experiments on two different platforms, a 3D physics engine with tens of agents as well as in a 2D grid world with up to thousands. The results clearly confirm our hypothesis and show substantial coordination both within and across species. To obtain these results, we leverage and adapt recent advances in deep reinforcement learning within an ecosystem training protocol featuring homogeneous groups of independent agents from different species (sets of policies), acting in many different random combinations in parallel habitats. The policies utilize neural network architectures that are invariant to agent individuality but not type (species) and that generalize across varying numbers of observed other agents. While the emergence of complexity in artificial ecosystems have long been studied in the artificial life community, the focus has been more on individual complexity and genetic algorithms or explicit modelling, and less on group complexity and reinforcement learning emphasized in this article. Unlike what the name and intuition suggests, reinforcement learning adapts over evolutionary history rather than a life-time and is here addressing the sequential optimization of fitness that is usually approached by genetic algorithms in the artificial life community. We utilize a shift from procedures to objectives, allowing us to bring new powerful machinery to bare, and we see emergence of complex behaviour from a sequence of simple optimization problems.

Cognition Evolves with the Emergence of Environmental Patchiness

“…If modeled prey were capable of movement (like fish, deer), active aggregation behaviors such as schooling and flocking could arise. Previous research has shown that flocking can emerge in simple predator-prey systems with limited sensation-action abilities (Sunehag et al, 2019). Our research makes a useful contribution in demonstrating how aggregation can emerge in prey without active aggregating behavior on their part, simply through the impact of evolved search patterns of predators.…”

The Ecosystem Path to General AI

“…For instance, they outperform humans at several video games and strategic board games [9]. There are also many agent-based ecosystem simulators that are based on RL [10,11,12].…”

dm_control: Software and tasks for continuous control

“…It is designed to facilitate the implementation of a wide range of locomotion tasks for RL algorithms by introducing self-contained, reusable components which compose into different task variants. The Locomotion framework has enabled a number of research efforts including [9][10][11] and more recently has been employed to support Multi-Agent domains in [12,13] and [14]. Fig.…”