On learning dynamics underlying the evolution of learning rules

Abstract

In order to understand the development of non-genetically encoded actions during an animal's lifespan, it is necessary to analyze the dynamics and evolution of learning rules producing behavior. Owing to the intrinsic stochastic and frequency-dependent nature of learning dynamics, these rules are often studied in evolutionary biology via agent-based computer simulations. In this paper, we show that stochastic approximation theory can help to qualitatively understand learning dynamics and formulate analytical models for the evolution of learning rules. We consider a population of individuals repeatedly interacting during their lifespan, and where the stage game faced by the individuals fluctuates according to an environmental stochastic process. Individuals adjust their behavioral actions according to learning rules belonging to the class of experience-weighted attraction learning mechanisms, which includes standard reinforcement and Bayesian learning as special cases. We use stochastic approximation theory in order to derive differential equations governing action play probabilities, which turn out to have qualitative features of mutator-selection equations. We then perform agent-based simulations to find the conditions where the deterministic approximation is closest to the original stochastic learning process for standard 2-action 2-player fluctuating games, where interaction between learning rules and preference reversal may occur. Finally, we analyze a simplified model for the evolution of learning in a producer-scrounger game, which shows that the exploration rate can interact in a non-intuitive way with other features of co-evolving learning rules. Overall, our analyses illustrate the usefulness of applying stochastic approximation theory in the study of animal learning.

A unified modelling approach for producer–scrounger games in complex ecological conditions

“…This model incorporates a commonly used learning rule as the individual's cognitive decision mechanism (Arbilly & Laland, 2014;Beauchamp, 2000;Beauchamp et al, 1997;Dridi & Lehmann, 2014;Dubois, Morand-Ferron, & Giraldeau, 2010;Hamblin & Giraldeau, 2009;Katsnelson, Motro, Feldman, & Lotem, 2012;Kurvers, Hamblin, & Giraldeau, 2012). We then tackle the second limitation by adding complexity to the environment beyond what has been possible with game-theoretic PS models and the experimental studies they generated.…”

“…Notwithstanding these difficulties concerning the origins of learning in groups, we choose to assume that our agents used learning rules to decide on tactic use, an assumption that is supported by a number of empirical studies. Learning rules, for instance, have been used in psychology, ethology and behavioural ecology for many years to model animal behaviour individually or in groups (Arbilly & Laland, 2014;Bush & Mosteller, 1951;Dridi & Lehmann, 2014;Estes, 1950;Hamblin & Giraldeau, 2009;Harley, 1981;Herrnstein, 1961;Krebs, Kacelnik, & Taylor, 1978;McNamara & Houston, 1985;Stephens, 1989). Moreover, in a series of producerescrounger experiments with nutmeg mannikins, Lonchura punctulata, Morand-Ferron and found that the group-level proportion of the join tactic adjusts to different environmental conditions by learning the payoffs associated with each tactic.…”

Why mutual helping in most natural systems is neither conflict-free nor based on maximal conflict

“…All we need to explain is the evolution of learning rules (like strength of reinforcement, weighing of past interactions, exploration of behavioural repertoire) that allow individuals to adjust their behaviour during their lifetime. Dridi & Lehmann [43,44] give wonderful examples of how exploratory trial-and-error reinforcement learning rules evolve that allow individuals to solve an iterated prisoner's dilemma.…”

“…Lack of mobility allows 'individual recognition' based on location, as is the case for ants interacting with their many partner species or for pollinators interacting with flowering plants. Moreover, memory capacities are apparently sufficient for learning through operant conditioning, with the evolution of crucial learning rule parameters subject to natural selection [43,44]. Indeed, it has repeatedly been argued that solving an iterated prisoner's dilemma game might not be as rare as often assumed because scientists were looking for strategies proposed by theoreticians, like tit-fortat, while animals make decisions differently.…”

“…One possible solution is to cooperate and to defect with probabilities that generate stable frequency dependence [102]. However, this will lead to cases in which both partners defect and hence both lose out, and hence various mechanisms may lead to the emergence of cooperators and defectors [43,72]. An even better solution would be that the two partners cooperate.…”