Back to optimality: a formal framework to express the dynamics of learning optimal behavior

Abstract

Citation: Alonso, E., Fairbank, M. & Mondragon, E. (2015). Back to optimality: a formal framework to express the dynamics of learning optimal behavior. Adaptive Behavior, 23(4), pp. 206-215. doi: 10.1177/1059712315589355 This is the accepted version of the paper.This version of the publication may differ from the final published version. Greetings, and thank you for publishing with SAGE. We have prepared this page proof for your review. Please respond to each of the below queries by digitally marking this PDF using Adobe Reader. PermanentClick "Comment" in the upper right corner of Adobe Reader to access the mark-up tools as follows:For textual edits, please use the "Annotations" tools. Please refrain from using the two tools crossed out below, as data loss can occur when using these tools.For formatting requests, questions, or other complicated changes, please insert a comment using "Drawing Markups." Abstract Whether animals behave optimally is an open question of great importance, both theoretically and in practice. Attempts to answer this question focus on two aspects of the optimization problem, the quantity to be optimized and the optimization process itself. In this paper, we assume the abstract concept of cost as the quantity to be minimized and propose a reinforcement learning algorithm, called Value-Gradient Learning (VGL), as a computational model of behavior optimality. We prove that, unlike standard models of Reinforcement Learning, Temporal Difference in particular, VGL is guaranteed to converge to optimality under certain conditions. The core of the proof is the mathematical equivalence of VGL and Pontryagin's Minimum Principle, a well-known optimization technique in systems and control theory. Given the similarity between VGL's formulation and regulatory models of behavior, we argue that our algorithm may provide psychologists with a tool to formulate such models in optimization terms.

A Reward-driven Model of Darwinian Fitness

“…Similarly, it is evident that, notwithstanding the accomplishments of computational theories of reinforcement learning in modelling neural and psychological factors (e.g., Dayan and Daw, 2008;Rangel et al, 2008;Schultz, 2008), the use of rewards in this area is a great simplification of the true nature of rewards . Although new ways to enrich the reinforcement learning ontology with ethological and evolutionary information have been reported (Alonso et al, 2015), the problem of integrating reward-driven approaches and fitness theories has not been tackled so far. It is apparent that whereas rewards reflect some fitness component, a general relationship between strength of rewards and fitness needs to be established.…”