Behavioral Diversity Generation in Autonomous Exploration through Reuse of Past Experience

Abstract

The production of behavioral diversity -producing a diversity of effects -is an essential strategy for robots exploring the world when facing situations where interaction possibilities are unknown or non-obvious. It allows to discover new aspects of the environment that cannot be inferred or deduced from available knowledge. However, creating behavioral diversity in situations where it is most crucial -new and unknown ones -is far from trivial. In particular in large and redundant sensorimotor spaces, only small areas are interesting to explore for any practical purpose. When the environment does not provide clues or gradient toward those areas, trying to discover those areas relies on chance. To address this problem, we introduce a method to create behavioral diversity in a new sensorimotor task by re-enacting actions that allowed to produce behavioral diversity in a previous task, along with a measure that quantifies this diversity. We show that our method can learn how to interact with an object by reusing experience from another, that it adapts to instances of morphological changes and of dissimilarity between tasks, and how scaffolding behaviors can emerge by simply switching the attention of the robot to different parts of the environment. Finally, we show that the method can robustly use simulated experiences and crude cognitive models to generate behavioral diversity in real robots.

Quality and Diversity Optimization: A Unifying Modular Framework

“…The concepts introduced with BR-Evolution have also later been employed in the Novelty-based Evolutionary Babbling (Nov-EB) [27] that allows a robot to autonomously discover the possible interactions with objects in its environment. This work draws a first link between the QD-algorithms and the domain of developmental robotics, which is also studied in several other works (see [28] for overview).…”

Intrinsic motivation, curiosity, and learning

“…Indeed, in many contexts, learning a single pre-defined skill can be difficult as it amounts to searching (the parameters of) a solution with very rare feedback until one is very close to the solution, or with deceptive feedback due to the phenomenon of local minima. A strategy to address these issues is to direct exploration with intrinsic rewards, leading the system to explore a diversity of skills and contingencies which often result in the discovery of new sub-spaces/areas in the problem space, or in mutual skill improvement when exploring one goal/skill provides data that can be used to improve other goals/skills, such as in goal babbling (Baranes and Oudeyer, 2013;Benureau and Oudeyer, 2016) or off-policy reinforcement learning (see the Horde architecture, Sutton et al, 2011). For example, Lehman and Stanley (2011) showed that searching for pure novelty in the behavioural space a robot to find a reward in a maze more efficiently than if it had been searching for behavioural parameters that optimized directly the reward.…”

Intrinsically Motivated Exploration of Learned Goal Spaces

“…At execution time, for a given goal τ , a loss function is defined over the parameterization space through L(θ) = C(τ, D(θ, c)). A black-box optimization algorithm, such as L-BFGS, is then used to optimize this function and find the optimal set of parameters θ (see [3,32,33] for examples of such meta-policy implementations in the IMGEP framework).…”