Adaptive bases for Q-learning

Castro, Dotan Di; Mannor, Shie

doi:10.1109/cdc.2010.5717385

Cited by 4 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A key contribution of this paper is a general framework for nonlinear value function approximation based on nonlinear separable least-squares. This framework helps clarify previously proposed algorithms (Castro and Mannor 2010;Bertsekas and Yu 2009;Menache, Shimkin, and Mannor 2005), which amount to particular instances of this general framework. Concretely, basis adaptation can be understood as modifying a parameterized basis, Φ(α) where α denotes the nonlinear parameters, and w denotes the linear parameters, such that V ≈ Φ(α)w. For example, for radial basis function (RBF) bases, α would correspond to the mean (center), μ i , and covariance (width) parameters, Σ i , of a fixed set of bases, where these would be tuned based on optimizing some particular error, such as the Bellman error (Menache, Shimkin, and Mannor 2005).…”

Section: Nonlinear Value Function Approximationmentioning

confidence: 82%

“…Like the approach proposed for adaptive bases for Qlearning (Castro and Mannor 2010), our method is based on the two-time scale stochastic approximation framework, whereby the linear parameters w are adapted at a faster timescale than the nonlinear parameters α. Algorithm 1 below describes the nonlinear adaptive basis mirror-descent variant of Watkins Q(λ) algorithm. 3 We indicate the dynamically varying nature of the bases as φ t (s t , a t ) where the subscript denotes the particular value of the nonlinear parameters α t at time t. In this section, we denote β t as the learning rate for the faster time-scale update procedure for updating the linear weights w t and ξ t as the learning rate for the slower time-scale parameter for updating the nonlinear basis parameters α t .…”

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

“…3 We indicate the dynamically varying nature of the bases as φ t (s t , a t ) where the subscript denotes the particular value of the nonlinear parameters α t at time t. In this section, we denote β t as the learning rate for the faster time-scale update procedure for updating the linear weights w t and ξ t as the learning rate for the slower time-scale parameter for updating the nonlinear basis parameters α t . Following the approach given in (Borkar 2008;Castro and Mannor 2010), it can be shown that Algorithm 1 converges broadly under similar assumptions. The key idea underlying Algorithm 1 is to adapt the nonlinear parameters using the gradient of the TD error.…”

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

“…The key idea underlying Algorithm 1 is to adapt the nonlinear parameters using the gradient of the TD error. As the gradient of the TD error is not easily computed due to the max computation in Q(λ), what is done here is to approximate the maximum using a smoothed differentiable approximation (Castro and Mannor 2010):…”

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

“…Basis construction algorithms (Mahadevan 2009) combine the learning of features, or basis functions, and control. Basis adaptation (Bertsekas and Yu 2009;Castro and Mannor 2010;Menache, Shimkin, and Mannor 2005) enables tuning a given parametric basis, such as the Fourier basis (Konidaris, Osentoski, and Thomas 2011), radial basis functions (RBF) (Menache, Shimkin, and Mannor 2005), polynomial bases (Lagoudakis and Parr 2003), etc. to the geometry of a particular Markov decision process (MDP).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Basis Adaptation for Sparse Nonlinear Reinforcement Learning

Mahadevan

Giguere

Jacek

2013

AAAI

View full text Add to dashboard Cite

This paper presents a new approach to representation discovery in reinforcement learning (RL) using basis adaptation. We introduce a general framework for basis adaptation as {\em nonlinear separable least-squares value function approximation} based on finding Frechet gradients of an error function using variable projection functionals. We then present a scalable proximal gradient-based approach for basis adaptation using the recently proposed mirror-descent framework for RL. Unlike traditional temporal-difference (TD) methods for RL, mirror descent based RL methods undertake proximal gradient updates of weights in a dual space, which is linked together with the primal space using a Legendre transform involving the gradient of a strongly convex function. Mirror descent RL can be viewed as a proximal TD algorithm using Bregman divergence as the distance generating function. We present a new class of regularized proximal-gradient based TD methods, which combine feature selection through sparse L1 regularization and basis adaptation. Experimental results are provided to illustrate and validate the approach.

show abstract

Section: Nonlinear Value Function Approximationmentioning

confidence: 82%

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

Section: Basis Adaptation With Mirror Descent Rlmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Basis Adaptation for Sparse Nonlinear Reinforcement Learning

Mahadevan

Giguere

Jacek

2013

AAAI

View full text Add to dashboard Cite

show abstract

Combined Sewer Overflow and Flooding Mitigation Through a Reliable Real‐Time Control Based on Multi‐Reinforcement Learning and Model Predictive Control

Tian

Liao

Zhi³

et al. 2022

Water Resources Research

View full text Add to dashboard Cite

Overflow and flooding can hardly be avoided in urban drainage system (UDS). Real-time control (RTC) has been proved effective to enhance the use of the existing systems for overflow and flooding mitigation (García et al., 2015;Kerkez et al., 2016;Schütze et al., 2002). Recently, a new RTC method based on reinforcement learning (RL) has been developed for flooding mitigation (Mullapudi et al., 2020;Saliba et al., 2020), achieving a milestone in the direction of smart water management. Despite the successful application of RL in UDS real-time control in modeling exercise, the effectiveness of different RLs in the context of UDS remains unclear. Meanwhile, the risk of handing over the control process to a RL agent is still unavoidable because of two reasons.First, the consequence of implementing a control strategy given by RL agent is unknown in practical application. While the value-based RLs use neural network to score their consequence (Mnih et al., 2015;Sutton & Barto, 2018), their neural network are black-box models which are insufficient to allow for a reliable evaluation. The methods of safe learning improve the reliability of RLs by reducing uncertainty, enhancing RL algorithm, and avoiding damage (Garcia & Fernández, 2015;Martin & Lope, 2009;Pal et al., 2019). But few of them forecast the consequence, and none of their methods is used in the RTC of UDS. Second, the nonlinearity of neural network causes fluctuation in the output of RL agent when facing some different inputs, and thus influence the performance of RL agent during controlling. This phenomenon also happens in other applications of neural network (Liu et al., 2021) and poses a certain risk in practical application. Therefore, forecasting the consequence of implementing RLs control strategies and reducing the influence of the fluctuation in the output of RL agents are necessary to improve the reliability of RLs.

show abstract

Hindsight States: Blending Sim & Real Task Elements for Efficient Reinforcement Learning

Guist,

Schneider,

Berenz

et al. 2023

Robotics: Science and Systems XIX

View full text Add to dashboard Cite

Reinforcement learning has shown great potential in solving complex tasks when large amounts of data can be generated with little effort. In robotics, one approach to generate training data builds on simulations based on dynamics models derived from first principles. However, for tasks that, for instance, involve complex soft robots, devising such models is substantially more challenging. Being able to train effectively in increasingly complicated scenarios with reinforcement learning enables to take advantage of complex systems such as soft robots. Here, we leverage the imbalance in complexity of the dynamics to learn more sample-efficiently. We (i) abstract the task into distinct components, (ii) off-load the simple dynamics parts into the simulation, and (iii) multiply these virtual parts to generate more data in hindsight. Our new method, Hindsight States (HiS), uses this data and selects the most useful transitions for training. It can be used with an arbitrary off-policy algorithm. We validate our method on several challenging simulated tasks and demonstrate that it improves learning both alone and when combined with an existing hindsight algorithm, Hindsight Experience Replay (HER). Finally, we evaluate HiS on a physical system and show that it boosts performance on a complex table tennis task with a muscular robot. Videos and code of the experiments can be found on webdav.tuebingen.mpg.de/his/.

show abstract

Adaptive bases for Q-learning

Cited by 4 publications

References 18 publications

Basis Adaptation for Sparse Nonlinear Reinforcement Learning

Basis Adaptation for Sparse Nonlinear Reinforcement Learning

Combined Sewer Overflow and Flooding Mitigation Through a Reliable Real‐Time Control Based on Multi‐Reinforcement Learning and Model Predictive Control

Hindsight States: Blending Sim & Real Task Elements for Efficient Reinforcement Learning

Contact Info

Product

Resources

About