Which is the best intrinsic motivation signal for learning multiple skills?

Abstract

Humans and other biological agents are able to autonomously learn and cache different skills in the absence of any biological pressure or any assigned task. In this respect, Intrinsic Motivations (i.e., motivations not connected to reward-related stimuli) play a cardinal role in animal learning, and can be considered as a fundamental tool for developing more autonomous and more adaptive artificial agents. In this work, we provide an exhaustive analysis of a scarcely investigated problem: which kind of IM reinforcement signal is the most suitable for driving the acquisition of multiple skills in the shortest time? To this purpose we implemented an artificial agent with a hierarchical architecture that allows to learn and cache different skills. We tested the system in a setup with continuous states and actions, in particular, with a kinematic robotic arm that has to learn different reaching tasks. We compare the results of different versions of the system driven by several different intrinsic motivation signals. The results show (a) that intrinsic reinforcements purely based on the knowledge of the system are not appropriate to guide the acquisition of multiple skills, and (b) that the stronger the link between the IM signal and the competence of the system, the better the performance.

Autonomous Reinforcement Learning of Multiple Interrelated Tasks

“…The learning progress in achieving a goal can be used as a transient reward so that the system focuses on tasks where it is learning the most, moving to other ones when the task-related skill has been completely learnt or when more promising activities come at hand [11], [12]. This strategy allows the learning of multiple separated skills, and possibly a dynamical transfer of knowledge between tasks that require similar policies [13].…”

Sensorimotor contingencies as a key drive of development: from babies to robots

“…The matching value is used to determine the probability distribution over the goals that supports the selection of the goal to pursue (number 2 in the figure). The probability distribution is generated on the basis of intrinsic motivations, for example related to competence (e.g., the goals with a lower competence, or with a higher competence-improvement, have a higher probability of selection; Santucci et al, 2013).…”

Toward Computational Motivation for Multi-Agent Systems and Swarms

“…Reinforcement Learning Barto et al, 2004;Simşek and Barto, 2006;Schembri et al, 2007;Sequeira et al, 2011;Kompella et al, 2012;Baldassarre and Mirolli, 2013;Metzen and Kirchner, 2013;Di Nocera et al, 2014;Frank et al, 2015;Hester and Stone, 2015 Deep Learning Mohamed and Rezende, 2015;Kulkarni et al, 2016;Achiam and Sastry, 2017;Zhelo et al, 2018 Hierarchical Structure Schembri et al, 2007;Baranes and Oudeyer, 2010;Baldassarre and Mirolli, 2013;Santucci et al, 2013;Frank et al, 2015;Kulkarni et al, 2016 Active Learning Baranes and Oudeyer, 2009, 2010Kompella et al, 2017;Pathak et al, 2017 Motion Planning Frank et al, 2015 Affordance Discovery Hart et al, 2008;Hart, 2009 Goal Discovery/Goal Generation In Reinforcement Learning (RL), an agent learns from experience as it deals with a sequential decision problem. The agent interacts with an "environment" which contains a "critic" that provides the agent with rewards by evaluating the behavior.…”