Aggregation Methods for Lineary-solvable Markov Decision Process

Zhong, Mingyuan; Todorov, Emanuel

doi:10.3182/20110828-6-it-1002.03729

Cited by 10 publications

(12 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Linearly-solvable optimal control is a rich mathematical framework that has recently received a lot of attention, following Kappen's work on control-affine diffusions in continuous time [14], and our work on Markov decision processes in discrete time [27]. Both groups have since then obtained many additional results: see [36,17,6,5] and [28,31,30,29,8,32,33,9,38,39] respectively. Other groups have also started to use and further develop this framework [35,7,24,25].…”

Section: Historical Perspectivementioning

confidence: 99%

Linearly Solvable Optimal Control

Dvijotham

Todorov

2012

Reinforcement Learning and Approximate Dynamic Programming for Feedback Control

View full text Add to dashboard Cite

We summarize the recently-developed framework of linearly-solvable stochastic optimal control. Using an exponential transformation, the (Hamilton-Jacobi) Bellman equation for such problems can be made linear, giving rise to efficient numerical methods. Extensions to game theory are also possible and lead to linear Isaacs equations.The key restriction that makes a stochastic optimal control problem linearly-solvable is that the noise and the controls must act in the same subspace. Apart from being linearly solvable, problems in this class have a number of unique properties including: path-integral interpretation of the exponentiated value function; compositionality of optimal control laws; duality with Bayesian inference; trajectory-based Maximum Principle for stochastic control. Development of a general class of more easily solvable problems tends to accelerate progress -as linear systems theory has done. The new framework may have similar impact in fields where stochastic optimal control is relevant.

show abstract

Section: Historical Perspectivementioning

confidence: 99%

Linearly Solvable Optimal Control

Dvijotham

Todorov

2012

Reinforcement Learning and Approximate Dynamic Programming for Feedback Control

View full text Add to dashboard Cite

show abstract

“…Recently, a new framework of linearly solvable Markov decision process (LMDP) has been proposed, in which a nonlinear Hamilton-Jacobi-Bellman (HJB) equation for continuous systems or Bellman equation for discrete systems is converted into a linear equation under certain assumptions on the action cost and the effect action on the state dynamics [6], [7]. In this approach, an exponentially transformed state value function is defined as a desirability function and it is derived from the linearized Bellman's equation by solving an eigenvalue problem [8] or an eigenfunction problem [9], [10]. One of the benefits is its compositionality.…”

Section: Introductionmentioning

confidence: 99%

Combining learned controllers to achieve new goals based on linearly solvable MDPs

Uchibe

Doya

2014

2014 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Learning complicated behaviors usually involves intensive manual tuning and expensive computational optimization because we have to solve a nonlinear HamiltonJacobi-Bellman (HJB) equation. Recently, Todorov proposed a class of the so-called Linearly solvable Markov Decision Process (LMDP) which converts a nonlinear HJB equation to a linear differential equation. Linearity of the simplified HJB equation allows us to apply superposition to derive a new composite controller from a set of learned primitive controllers. However, his method was a model-based approach and it was not evaluated in a real domain. This study proposes a modelfree method which is similar to the Least Squares Temporal Difference (LSTD) learning. In this method, the exponentially transformed cost function can be regarded as the discount factor in LSTD. Our proposed method is applied to learning walking behaviors with the quadruped robot to evaluate in real robot experiments. The goal of each primitive task is to go to the specific target position in the environment and that of the composite task is to approach arbitrary region represented by the primitives' target positions. Experimental results show that the composite policy can be used as a good initial policy for the new task.

show abstract

“…However, an additional learning is needed when a new initial state or a new goal state is given. In the value-based approach, an exponentially transformed state value function is defined as the desirability function and it is derived from the linearized Bellman's equation by solving an eigenvalue problem (Todorov, 2007) or an eigenfunction problem (Todorov, 2009c; Zhong and Todorov, 2011). One of the benefits of the desirability function approach is its compositionality.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

Kinjo¹,

Uchibe²,

Doya³

2013

Front. Neurorobot.

View full text Add to dashboard Cite

Linearly solvable Markov Decision Process (LMDP) is a class of optimal control problem in which the Bellman's equation can be converted into a linear equation by an exponential transformation of the state value function (Todorov, 2009b). In an LMDP, the optimal value function and the corresponding control policy are obtained by solving an eigenvalue problem in a discrete state space or an eigenfunction problem in a continuous state using the knowledge of the system dynamics and the action, state, and terminal cost functions. In this study, we evaluate the effectiveness of the LMDP framework in real robot control, in which the dynamics of the body and the environment have to be learned from experience. We first perform a simulation study of a pole swing-up task to evaluate the effect of the accuracy of the learned dynamics model on the derived the action policy. The result shows that a crude linear approximation of the non-linear dynamics can still allow solution of the task, despite with a higher total cost. We then perform real robot experiments of a battery-catching task using our Spring Dog mobile robot platform. The state is given by the position and the size of a battery in its camera view and two neck joint angles. The action is the velocities of two wheels, while the neck joints were controlled by a visual servo controller. We test linear and bilinear dynamic models in tasks with quadratic and Guassian state cost functions. In the quadratic cost task, the LMDP controller derived from a learned linear dynamics model performed equivalently with the optimal linear quadratic regulator (LQR). In the non-quadratic task, the LMDP controller with a linear dynamics model showed the best performance. The results demonstrate the usefulness of the LMDP framework in real robot control even when simple linear models are used for dynamics learning.

show abstract

Aggregation Methods for Lineary-solvable Markov Decision Process

Cited by 10 publications

References 2 publications

Linearly Solvable Optimal Control

Linearly Solvable Optimal Control

Combining learned controllers to achieve new goals based on linearly solvable MDPs

Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

Contact Info

Product

Resources

About