Hierarchical Affordance Discovery using Intrinsic Motivation

Abstract

To be capable of lifelong learning in a real-life environment, robots have to tackle multiple challenges. Being able to relate physical properties they may observe in their environment to possible interactions they may have is one of them. This skill, named affordance learning, is strongly related to embodiment and is mastered through each person's development: each individual learns affordances differently through their own interactions with their surroundings. Current methods for affordance learning usually use either fixed actions to learn these affordances or focus on static setups involving a robotic arm to be operated. In this article, we propose an algorithm using intrinsic motivation to guide the learning of affordances for a mobile robot. This algorithm is capable to autonomously discover, learn and adapt interrelated affordances without pre-programmed actions. Once learned, these affordances may be used by the algorithm to plan sequences of actions in order to perform tasks of various difficulties. We then present one experiment and analyse our system before comparing it with other approaches from reinforcement learning and affordance learning.

Autonomous Reinforcement Learning of Multiple Interrelated Tasks

“…The concept of Intrinsic Motivations (IMs) is borrowed from biological [9] and psychological literature [10] describing how novel or unexpected "neutral" stimuli, as well as the perception of control over the environment, can generate learning processes even in the absence of assigned rewards or tasks. In the computational literature, IMs have been implemented in artificial agents to foster their autonomy in gathering knowledge [11], [12], learning repertoire of skills [13], [14], [15], [16], exploiting affordances from the environment [17], [18], [19], selecting their own tasks [20], [21], [22], and even boosting imitation learning techniques [23].…”

Interdependence as the key for an ethical artificial autonomy

“…The concept of intrinsic motivations (IMs) is borrowed from the literature on animals (White 1959) and from human psychology (Ryan and Deci 2000), describing how novel or unexpected 'neutral' stimuli, as well as the perception of control over the environment, determine learning processes even in the absence of assigned rewards or goals. In the computational literature, IMs have been implemented to foster different types of autonomous behaviors such as state-space exploration (Bellemare et al 2016;Romero et al 2020;Schillaci et al 2020), knowledge gathering (Schmidhuber 2010), learning repertoire of skills (Singh et al 2004;Oudeyer et al 2013), affordance exploration and exploitation (Hart and Grupen 2013;Baldassarre et al 2019;Manoury et al 2019). Furthermore, and closely related to the topic discussed in this article, IMs have been used to allow embodied artificial agents to autonomously discover and select their own goals (Baranes and Oudeyer 2013;Santucci et al 2016Santucci et al , 2019Blaes et al 2020).…”

Intrinsically Motivated Open-Ended Multi-Task Learning Using Transfer Learning to Discover Task Hierarchy

“…More recently, Intrinsic Motivation has also tacked hierarchical RL to build increasingly complex skills by discovering and exploiting the task hierarchy using planning methods [25]. However it does not model explicitly a representation of the task hierarchy, letting planning compose the sequences in the exploitation phase.…”

“…Our work proposes a representation of the relationship between tasks and their complexities, by the procedural framework. Comparatively, [25] proposed for tackling a hierarchical multi-task setting, to learn action primitives and use planning to recursively chain skills. However, that approach does not build a representation of a sequence of action primitives, and planning grows slower as the environment is explored.…”