Learning state representations with robotic priors

Jonschkowski, Rico; Brock, Oliver

doi:10.1007/s10514-015-9459-7

Cited by 131 publications

(158 citation statements)

References 27 publications

Supporting

Mentioning

150

Contrasting

Order By: Relevance

“…In summary, our information-theoretic principle (8) for perception-action coupling provides a novel generic principled method and could in principle be applied to combine any parameterized perception and action modules. Compared to the existing literature, this approach is most similar in spirit to approaches that learn particular perceptual features that are most useful to solve a particular task [40], [41], [42], [43]. Here this feature search is integrated in a single bounded rational optimization problem.…”

Section: Discussionmentioning

confidence: 99%

An information-theoretic on-line update principle for perception-action coupling

Peng

Genewein

Leibfried³

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Inspired by findings of sensorimotor coupling in humans and animals, there has recently been a growing interest in the interaction between action and perception in robotic systems [1]. Here we consider perception and action as two serial information channels with limited information-processing capacity. We follow [2] and formulate a constrained optimization problem that maximizes utility under limited informationprocessing capacity in the two channels. As a solution we obtain an optimal perceptual channel and an optimal action channel that are coupled such that perceptual information is optimized with respect to downstream processing in the action module. The main novelty of this study is that we propose an online optimization procedure to find bounded-optimal perception and action channels in parameterized serial perception-action systems. In particular, we implement the perceptual channel as a multi-layer neural network and the action channel as a multinomial distribution. We illustrate our method in a NAO robot simulator with a simplified cup lifting task.

show abstract

Section: Discussionmentioning

confidence: 99%

An information-theoretic on-line update principle for perception-action coupling

Peng

Genewein

Leibfried³

et al. 2017

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…To address this challenge, a potential starting point is to use unsupervised learning to learn low-dimensional features, which can then be used as inputs for policies. Interestingly, it is possible to leverage priors to learn such state representations from raw observations in a reasonable interaction time [85,112]. It is also possible to create forward models in image space, that is, predicting the next image knowing the current one and the actions, which would allow to design model-based policy search algorithms that work with an image stream [9,52,63,137].…”

Section: Scalabilitymentioning

confidence: 99%

“…More generic or task-agnostic priors (e.g., properties of the physical world) could relax these assumptions while still providing a learning speedup. Some steps have been made into identifying such task-agnostic priors for robotics, and using them for state representation [85,111]. We believe this is an important direction that requires more investigation.…”

Section: Priorsmentioning

confidence: 99%

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

et al. 2020

View full text Add to dashboard Cite

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word "big-data", we refer to this challenge as "micro-data reinforcement learning". We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots, designing generic priors, and optimizing the computing time.1 In some rare cases, a process can be "optimally efficient". 2 It is challenging to put a precise limit for "micro-data learning" as each domain has different experimental constraints, this is why we will refer in this article to "a few minutes" or a "a few trials". The commonly used word "big-data" has a similar "fuzzy" limit that depends on the exact domain. 3 Planning-based and model-predictive control [59] methods do not search for policy parameters, this is why they do not fit into the scope of this paper. Although trajectory-based policies and planning-based methods share the same goal, they usually search in a different space: planning algorithms search in the state-action space (e.g., joint positions/velocities), whereas policy methods will search for the optimal parameters of the policy, which can encode a Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 1 arXiv:1807.02303v4 [cs.RO] 31 May 2019 Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 2This is basically sampling the distribution over trajectories, P (τ |θ), which is feasible since the sampling is performed with the models. When applying the same policy (i.e., a policy with the same parameters θ), the trajectories τ (and consequently r)Chatzilygeroudis, Vassiliades, Stulp, Calinon and Mouret arXiv | 9

show abstract

“…To address Challenge 1, state representations can be learned from known actions (Jonschkowski and Brock, 2015) and, likewise, to address Challenge 2, actions can be learned when the state space is known (Rolf et al, 2010; Forestier et al, 2017). …”

Section: Challengesmentioning

confidence: 99%

Open-Ended Learning: A Conceptual Framework Based on Representational Redescription

et al. 2018

View full text Add to dashboard Cite

Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.

show abstract

Learning state representations with robotic priors

Cited by 131 publications

References 27 publications

An information-theoretic on-line update principle for perception-action coupling

An information-theoretic on-line update principle for perception-action coupling

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

Open-Ended Learning: A Conceptual Framework Based on Representational Redescription

Contact Info

Product

Resources

About