Novelty Detection as an Intrinsic Motivation for Cumulative Learning Robots

Nehmzow, Ulrich; Gatsoulis, Yiannis; Kerr, Emmett; Condell, Joan; Siddique, Nazmul; McGuinnity, T. Martin

doi:10.1007/978-3-642-32375-1_8

Cited by 16 publications

(19 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…I have argued that the theory explains essential aspects of intelligence including selective attention, curiosity, creativity, science, art, music, humor, e.g., (Schmidhuber, 2006a, 2010). Compare recent related work, e.g., (Salge et al, 2012; Barto, 2013; Dayan, 2013; Nehmzow et al, 2013; Oudeyer et al, 2013). …”

Section: Discussionsupporting

confidence: 59%

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

Schmidhuber¹

2013

Front. Psychol.

107

View full text Add to dashboard Cite

Most of computer science focuses on automatically solving given computational problems. I focus on automatically inventing or discovering problems in a way inspired by the playful behavior of animals and humans, to train a more and more general problem solver from scratch in an unsupervised fashion. Consider the infinite set of all computable descriptions of tasks with possibly computable solutions. Given a general problem-solving architecture, at any given time, the novel algorithmic framework PowerPlay (Schmidhuber, 2011) searches the space of possible pairs of new tasks and modifications of the current problem solver, until it finds a more powerful problem solver that provably solves all previously learned tasks plus the new one, while the unmodified predecessor does not. Newly invented tasks may require to achieve a wow-effect by making previously learned skills more efficient such that they require less time and space. New skills may (partially) re-use previously learned skills. The greedy search of typical PowerPlay variants uses time-optimal program search to order candidate pairs of tasks and solver modifications by their conditional computational (time and space) complexity, given the stored experience so far. The new task and its corresponding task-solving skill are those first found and validated. This biases the search toward pairs that can be described compactly and validated quickly. The computational costs of validating new tasks need not grow with task repertoire size. Standard problem solver architectures of personal computers or neural networks tend to generalize by solving numerous tasks outside the self-invented training set; PowerPlay’s ongoing search for novelty keeps breaking the generalization abilities of its present solver. This is related to Gödel’s sequence of increasingly powerful formal theories based on adding formerly unprovable statements to the axioms without affecting previously provable theorems. The continually increasing repertoire of problem-solving procedures can be exploited by a parallel search for solutions to additional externally posed tasks. PowerPlay may be viewed as a greedy but practical implementation of basic principles of creativity (Schmidhuber, 2006a, 2010). A first experimental analysis can be found in separate papers (Srivastava et al., 2012a,b, 2013).

show abstract

Section: Discussionsupporting

confidence: 59%

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

Schmidhuber¹

2013

Front. Psychol.

107

View full text Add to dashboard Cite

show abstract

“…If none of the first four outputs exceeds the threshold, the fovea does not move. Similarily, the next four outputs are interpreted as the extent of movement of the fovea over the image (3,9,27 or 81 pixels in case of exceeding the threshold, 1 pixel otherwise).…”

Section: B Rnn-controlled Fovea Environmentmentioning

confidence: 99%

“…[1], [4], [11], [9]), POWERPLAY provably (by design) does not have any problems with online learning-it cannot forget previously learned skills, automatically segmenting its life into a sequence of clearly identified tasks with explicitly recorded solutions. Unlike the task search of theoretically optimal creative agents [20], [21], POWERPLAY's task search is greedy, yet practically feasible.…”

Section: Introductionmentioning

confidence: 99%

Continually adding self-invented problems to the repertoire: First experiments with POWERPLAY

Srivastava

Steunebrink

Stollenga

et al. 2012

2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL)

View full text Add to dashboard Cite

Abstract-Pure scientists do not only invent new methods to solve given problems. They also invent new problems. The recent POWERPLAY framework formalizes this type of curiosity and creativity in a new, general, yet practical way. To acquire problem solving prowess through playing, POWERPLAY-based artificial explorers by design continually come up with the fastest to find, initially novel, but eventually solvable problems. They also continually simplify or speed up solutions to previous problems. We report on results of first experiments with POWERPLAY. A self-delimiting recurrent neural network (SLIM RNN) is used as a general computational architecture to implement the system's solver. Its weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences. In open-ended fashion, our POWERPLAY-driven RNNs learn to become increasingly general problem solvers, continually adding new problem solving procedures to the growing repertoire, exhibiting interesting developmental stages.

show abstract

“…Unlike our first implementations of curious / creative / playful agents from the 1990s [17,29,18] (cf. [1,4,13,11]), POWERPLAY provably (by design) does not have any problems with online learning-it cannot forget previously learned skills, automatically segmenting its life into a sequence of clearly identified tasks with explicitly recorded solutions. Unlike the task search of theoretically optimal creative agents [22,23], POWERPLAY's task search is greedy, yet practically feasible.…”

Section: Introductionmentioning

confidence: 99%

First experiments with PowerPlay

2013

View full text Add to dashboard Cite

Like a scientist or a playing child, POWERPLAY (Schmidhuber, 2011) not only learns new skills to solve given problems, but also invents new interesting problems by itself. By design, it continually comes up with the fastest to find, initially novel, but eventually solvable tasks. It also continually simplifies or compresses or speeds up solutions to previous tasks. Here we describe first experiments with POWERPLAY. A self-delimiting recurrent neural network SLIM RNN (Schmidhuber, 2012) is used as a general computational problem solving architecture. Its connection weights can encode arbitrary, self-delimiting, halting or non-halting programs affecting both environment (through effectors) and internal states encoding abstractions of event sequences. Our POWERPLAY-driven SLIM RNN learns to become an increasingly general solver of self-invented problems, continually adding new problem solving procedures to its growing skill repertoire. Extending a recent conference paper (Srivastava, Steunebrink, Stollenga, & Schmidhuber, 2012), we identify interesting, emerging, developmental stages of our open-ended system. We also show how it automatically self-modularizes, frequently re-using code for previously invented skills, always trying to invent novel tasks that can be quickly validated because they do not require too many weight changes affecting too many previous tasks.

show abstract

Novelty Detection as an Intrinsic Motivation for Cumulative Learning Robots

Cited by 16 publications

References 42 publications

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

Continually adding self-invented problems to the repertoire: First experiments with POWERPLAY

First experiments with PowerPlay

Contact Info

Product

Resources

About