What is intrinsic motivation? A typology of computational approaches

Abstract

Intrinsic motivation, centrally involved in spontaneous exploration and curiosity, is a crucial concept in developmental psychology. It has been argued to be a crucial mechanism for open-ended cognitive development in humans, and as such has gathered a growing interest from developmental roboticists in the recent years. The goal of this paper is threefold. First, it provides a synthesis of the different approaches of intrinsic motivation in psychology. Second, by interpreting these approaches in a computational reinforcement learning framework, we argue that they are not operational and even sometimes inconsistent. Third, we set the ground for a systematic operational study of intrinsic motivation by presenting a formal typology of possible computational approaches. This typology is partly based on existing computational models, but also presents new ways of conceptualizing intrinsic motivation. We argue that this kind of computational typology might be useful for opening new avenues for research both in psychology and developmental robotics.Keywords: intrinsic motivation, cognitive development, reward, reinforcement learning, exploration, curiosity, computational modeling, artifi cial intelligence, developmental robotics INTRODUCTIONThere exists a wide diversity of motivation systems in living organisms, and humans in particular. For example, there are systems that push the organism to maintain certain levels of chemical energy, involving the ingestion of food, or systems that push the organism to maintain its temperature or its physical integrity in a zone of viability. Inspired by these kinds of motivation and their understanding by (neuro-) ethologists, roboticists have built machines endowed with similar systems with the aim of providing them with autonomy and properties of life-like intelligence (Arkin, 2005). For example sowbug-inspired robots (Endo and Arkin, 2001), praying mantis robots (Arkin et al., 1998) dog-like robots (Fujita et al., 2001) have been constructed. Some animals, and this is most prominent in humans, also have more general motivations that push them to explore, manipulate or probe their environment, fostering curiosity and engagement in playful and new activities. This kind of motivation, which is called intrinsic motivation by psychologists (Ryan and Deci, 2000), is paramount for sensorimotor and cognitive development throughout lifespan. There is a vast literature in psychology that explains why it is essential for cognitive growth and organization, and investigates the actual potential cognitive processes underlying intrinsic motivation (Berlyne, 1960;Csikszentmihalyi, 1991;Deci and Ryan, 1985;Ryan and Deci, 2000;White, 1959). This has gathered the interest of a growing number of researchers in developmental robotics in the recent years, and several computational models have been developed (see Barto et al., 2004; Oudeyer et al., 2007 for reviews).However, the very concept of intrinsic motivation has never really been consistently and critically discussed from a computational point ...

Behavioral Diversity Generation in Autonomous Exploration through Reuse of Past Experience

“…Goal-directed exploration (Oudeyer and Kaplan, 2007;Baranes and Oudeyer, 2010;Jamone et al, 2011;Rolf et al, 2011;Baranes and Oudeyer, 2013), as well as related methods such as MAPElite (Cully et al, 2015), has been shown to be effective at creating behavioral diversity in large sensorimotor spaces. However, these methods only consider a single task.…”

“…Goal-directed exploration (Oudeyer and Kaplan, 2007) implementations have been proposed in Baranes and Oudeyer (2010), Jamone et al (2011), andRolf et al (2011), and as part of the SAGG-RIAC architecture (Baranes and Oudeyer, 2013) and have been shown to be effective in exploring sensorimotor spaces with large motor spaces. These methods for goal babbling as well as related methods such as MAP-Elite (Cully et al, 2015) have also been shown to efficiently generate forms of behavioral diversity.…”

“…Instead, it creates a set of observations about diverse features of the environment, offering the robot a set of options that can be explored and exploited toward specific objectives afterward. The usefulness of such strategies for robots has been recently explored through models of curiosity-driven learning and intrinsically motivated reinforcement learning (Oudeyer and Kaplan, 2007;Baldassarre and Mirolli, 2013;Benureau and Oudeyer, 2015), and in a related line of work, on novelty and diversity search in evolutionary robotics Lehman and Stanley, 2011).…”

Active Control for Object Perception and Exploration with a Robotic Hand

“…In this work, we use a predictive novelty motivation model, where interesting locations for exploration are those for which prediction errors are higher [12]. This is defined as follows:…”

“…This approach studied by psychology and cognitive sciences [9], [10], have found that intrinsic motivation is essential for cognitive development in humans, and also required for robust exploration and manipulation in robotics [11], [12], [13].…”