Sequential Constant Size Compressors for Reinforcement Learning

The Frontiers Collection

2012

Self Cite

Summary. Artificial Intelligence (AI) has recently become a real formal science: the new millennium brought the first mathematically sound, asymptotically optimal, universal problem solvers, providing a new, rigorous foundation for the previously largely heuristic field of General AI and embedded agents. There also has been rapid progress in not quite universal but still rather general and practical artificial recurrent neural networks for learning sequenceprocessing programs, now yielding state-of-the-art results in real world applications. And the computing power per Euro is still growing by a factor of 100-1000 per decade, greatly increasing the feasibility of neural networks in general, which have started to yield humancompetitive results in challenging pattern recognition competitions. Finally, a recent formal theory of fun and creativity identifies basic principles of curious and creative machines, laying foundations for artificial scientists and artists. Here I will briefly review some of the new results of my lab at IDSIA, and speculate about future developments, pointing out that the time intervals between the most notable events in over 40,000 years or 2 9 lifetimes of human history have sped up exponentially, apparently converging to zero within the next few decades. Or is this impression just a by-product of the way humans allocate memory space to past events? Note: this is the 2012 update of a 2007 publication [73]. Compare also the 2006 celebration of 75 years of AI [71].

Section: Recurrent / Deep Neural Networkmentioning

confidence: 99%

New Millennium AI and the Convergence of History: Update of 2012

The Frontiers Collection

2012

Self Cite

“…TD [23], policy gradients [22], etc. ), is to combine action learning with an unsupervised learning (UL) preprocessor or "compressor" which provides a lower-dimensional feature vector that the agent receives as input instead of the raw observation [5,8,11,15,17,18,19]. The UL compressor is trained on the high-dimensional observations generated by the learning agent's actions, that the agent then uses as a state representation to learn a value function.…”

Section: Introductionmentioning

confidence: 99%

Evolving deep unsupervised convolutional networks for vision-based reinforcement learning

Koutník

Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation

Gomez

2014

114

Dealing with high-dimensional input spaces, like visual input, is a challenging task for reinforcement learning (RL). Neuroevolution (NE), used for continuous RL problems, has to either reduce the problem dimensionality by (1) compressing the representation of the neural network controllers or (2) employing a pre-processor (compressor) that transforms the high-dimensional raw inputs into low-dimensional features. In this paper, we are able to evolve extremely small recurrent neural network (RNN) controllers for a task that previously required networks with over a million weights. The high-dimensional visual input, which the controller would normally receive, is first transformed into a compact feature vector through a deep, max-pooling convolutional neural network (MPCNN). Both the MPCNN preprocessor and the RNN controller are evolved successfully to control a car in the TORCS racing simulator using only visual input. This is the first use of deep learning in the context evolutionary RL.

“…Up to now, this general approach of using UL as a preprocessor has been studied only in the context of single-agent RL (i.e. TD, policy gradients, etc), where the agent simultaneously learns both the mapping from the compressed features to actions, and the features themselves from the observations provoked by the actions [1,4,11,13,14,16]. A problem with this coupled system is that, since the learned features depend on how the agent behaves and vice-versa, good features may never form because the agent's initial, poor policy results in biased sampling of the environment.…”

Section: Introductionmentioning

confidence: 99%

Intrinsically motivated neuroevolution for vision-based reinforcement learning

Cuccu

Luciw

2011 IEEE International Conference on Development and Learning (ICDL)

et al. 2011

Abstract-Neuroevolution, the artificial evolution of neural networks, has shown great promise on continuous reinforcement learning tasks that require memory. However, it is not yet directly applicable to realistic embedded agents using high-dimensional (e.g. raw video images) inputs, requiring very large networks. In this paper, neuroevolution is combined with an unsupervised sensory pre-processor or compressor that is trained on images generated from the environment by the population of evolving recurrent neural network controllers. The compressor not only reduces the input cardinality of the controllers, but also biases the search toward novel controllers by rewarding those controllers that discover images that it reconstructs poorly. The method is successfully demonstrated on a vision-based version of the wellknown mountain car benchmark, where controllers receive only single high-dimensional visual images of the environment, from a third-person perspective, instead of the standard two-dimensional state vector which includes information about velocity.