Seeking entropy: complex behavior from intrinsic motivation to occupy action-state path space

Abstract

Intrinsic motivation generates behaviors that do not necessarily lead to immediate reward, but help exploration and learning. Here we show that agents having the sole goal of maximizing occupancy of future actions and states, that is, moving and exploring on the long term, are capable of complex behavior without any reference to external rewards. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy with the optimal state-value function, from where we prove uniqueness of the solution of the associated Bellman equation and convergence of our algorithm to the optimal state-value function. Using discrete and continuous state tasks, we show that 'dancing', hide-and-seek and a basic form of altruistic behavior naturally result from entropy seeking without external rewards. Intrinsically motivated agents can objectively determine what states constitute rewards, exploiting them to ultimately maximize action-state path entropy.

Beyond simple laboratory studies: developing sophisticated models to study rich behavior

“…Another study that used a similar approach revealed that people prefer options that make more future options available, above and beyond utility concerns; and that the availability of options positively influenced participants' perceived freedom of choice [128]. This resonates with findings from a recent model based on an information theoretic approach [129]. This study shows that endowing agents with the tendency to maximize the foreseen occupancy of their future actions and states can account for a range of complex behaviors with no need to invoke reward seeking.…”

Fluctuations in sequential many-alternative decisions reveal strategies beyond immediate reward maximisation

“…Individual features such as risk aversion (third term in Eq.1) have been demonstrated to influence the allocation of limited resources in various types of uncertain decisions (Chronopoulos et al, 2011;Dow & Werlang, 1992;Tulloch et al, 2015), suggesting that it might affect participants' propensity to skip a trial ( ) and leave it to chance. Finally, participants sampling behaviour may be driven by a tendency to occupy action-state space (maximum occupancy principle), compelling them to try out various resources allocations and gain a global understanding of the environment (Ramírez-Ruiz et al, 2022). To model this, we introduced an entropy term.…”

“…Such behavioural tendencies contribute to reducing prediction errors and constructing a more accurate model of the environment, facilitating rapid adaptation to potential changes. Moreover, the maximum occupancy principle, which favours entropy seeking, constitutes the core objective of newer theoretical frameworks modelling behaviour, offering an alternative perspective to reward maximisation (Ramírez-Ruiz et al, 2022).…”

“…Furthermore, we explore the specific contexts in which these fluctuations transpire, considering factors such as available capacity, previous and future resource allocation and reward history. In the final part of our study, we extend the optimal model to investigate the potential presence of underlying strategies, entropy seeking from the maximum occupancy principle (Ramírez-Ruiz et al, 2022), risk aversion (Tulloch et al, 2015) and the pursuit of relevant information (Gottlieb, 2012), that elucidate the intentionality behind the observed increased variability.…”