Distributional Reinforcement Learning with Quantile Regression

Dabney, Will; Rowland, Mark; Bellemare, Marc G.; Munos, Rémi

doi:10.48550/arxiv.1710.10044

Cited by 17 publications

(33 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address the issue of risk in RL, recent works [6], [7] have introduced the concept of distributional RL. Distributional RL learns the distribution of accumulated rewards, Fig.…”

Section: Imentioning

confidence: 99%

“…Empirically, distributional RL algorithms have shown superior performance and sample efficiency in many game domains [7], [36]. It has been argued that this is because predicting quantiles serves as an auxiliary task that enhances representation learning, but as yet there is little supporting evidence for this conjecture [12].…”

Section: Distributional Rl and Risk-sensitive Policiesmentioning

confidence: 99%

“…2) Distributional SAC and Risk-Sensitive Policies: To capture the full distribution of accumulated rewards, rather than just its mean, [38] recently proposed distributional SAC (DSAC). As in previous work on distributional RL [7], [8], [37], DSAC uses quantile regression to learn this distribution. Such previous work is limited to finite action spaces, and DSAC overcomes this limitation using ideas from SAC.…”

Section: B Risk-conditioned Distributional Soft Actor-criticmentioning

confidence: 99%

See 2 more Smart Citations

Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation

Choi¹,

Dance²,

Kim³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern navigation algorithms based on deep reinforcement learning (RL) show promising efficiency and robustness. However, most deep RL algorithms operate in a riskneutral manner, making no special attempt to shield users from relatively rare but serious outcomes, even if such shielding might cause little loss of performance. Furthermore, such algorithms typically make no provisions to ensure safety in the presence of inaccuracies in the models on which they were trained, beyond adding a cost-of-collision and some domain randomization while training, in spite of the formidable complexity of the environments in which they operate. In this paper, we present a novel distributional RL algorithm that not only learns an uncertainty-aware policy, but can also change its risk measure without expensive fine-tuning or retraining. Our method shows superior performance and safety over baselines in partiallyobserved navigation tasks. We also demonstrate that agents trained using our method can adapt their policies to a wide range of risk measures at run-time.

show abstract

“…To address the issue of risk in RL, recent works [6], [7] have introduced the concept of distributional RL. Distributional RL learns the distribution of accumulated rewards, Fig.…”

Section: Imentioning

confidence: 99%

Section: Distributional Rl and Risk-sensitive Policiesmentioning

confidence: 99%

Section: B Risk-conditioned Distributional Soft Actor-criticmentioning

confidence: 99%

See 1 more Smart Citation

Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation

Choi¹,

Dance²,

Kim³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We propose several Reverse RL algorithms and prove their convergence under linear function approximation. We also propose Distributional Reverse RL algorithms akin to Distributional RL (Bellemare et al, 2017;Dabney et al, 2017;Rowland et al, 2018) to compute the probability of an event for anomaly detection. We demonstrate empirically the utility of Reverse GVFs in anomaly detection and representation learning.…”

Section: L1 L2 L4 L3mentioning

confidence: 99%

“…We now provide a practical algorithm to approximate η s π based on quantile regression, akin to Dabney et al (2017). We use N quantiles with quantile levels {τ i } i=1,...,N , where τ i .…”

Section: Reverse General Value Functionmentioning

confidence: 99%

Learning Retrospective Knowledge with Reverse Reinforcement Learning

Zhang¹,

Veeriah²,

Whiteson³

2020

Preprint

View full text Add to dashboard Cite

We present a Reverse Reinforcement Learning (Reverse RL) approach for representing retrospective knowledge. General Value Functions (GVFs) have enjoyed great success in representing predictive knowledge, i.e., answering questions about possible future outcomes such as "how much fuel will be consumed in expectation if we drive from A to B?". GVFs, however, cannot answer questions like "how much fuel do we expect a car to have given it is at B at time t?". To answer this question, we need to know when that car had a full tank and how that car came to B. Since such questions emphasize the influence of possible past events on the present, we refer to their answers as retrospective knowledge. In this paper, we show how to represent retrospective knowledge with Reverse GVFs, which are trained via Reverse RL. We demonstrate empirically the utility of Reverse GVFs in both representation learning and anomaly detection.

show abstract

Artificial Intelligence for Prosthetics: Challenge Solutions

Kidziński

Ong

Mohanty

et al. 2019

The Springer Series on Challenges in Machine Learning

View full text Add to dashboard Cite

In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector. Top participants were invited to describe their algorithms. In this work, we describe the challenge and present thirteen solutions that used deep reinforcement learning approaches. Many solutions use similar relaxations and heuristics, such as reward shaping, frame skipping, discretization of the action space, symmetry, and policy blending. However, each team implemented different modifications of the known algorithms by, for example, dividing the task into subtasks, learning low-level control, or by incorporating expert knowledge and using imitation learning.

show abstract

Distributional Reinforcement Learning with Quantile Regression

Cited by 17 publications

References 0 publications

Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation

Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation

Learning Retrospective Knowledge with Reverse Reinforcement Learning

Artificial Intelligence for Prosthetics: Challenge Solutions

Contact Info

Product

Resources

About