Having “multiple selves” helps learning agents explore and adapt in complex changing worlds

Abstract

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of sub-agents, each dedicated to a separate need, powerfully enhanced the agent's capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multi-objective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents: a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in non-stationary environments, and c) scaled gracefully in their ability to maintain homeostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of 'multiple selves'.

Adapting to loss: A normative account of grief

“…Lastly, it will be important to scale our results up to more complex, ecologically valid environments. Modular deep Q-learning is a good candidate model for this (Dulberg, Dubey, Berwian, & Cohen, 2023), since modules maintain different memory buffers corresponding to separate reward components. This architecture could naturally accommodate memory re-labelling of distinct sources of value.…”