Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Dorner, Florian E.

doi:10.48550/arxiv.2102.04881

Cited by 3 publications

(5 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sample efficiency (SE) (Chen et al, 2021 ; Dorner, 2021 ) is measured by the ratio of the number of samples collected when RAC and some algorithms reach the specified performance. Hopper is not in the comparison object as the performance of algorithms is almost indistinguishable.…”

Section: Methodsmentioning

confidence: 99%

“…Results of MBPO are obtained at 3 × 10 5 time steps for Ant, Humanoid, and Walker2d, 4 × 10 5 for HalfCheetah and 1.25 × 10 5 for Hopper. Sample efficiency (Chen et al, 2021;Dorner, 2021) is measured by the ratio of the number of samples collected when RAC and some algorithms reach the specified performance. The last four rows show how many times RAC is more sample efficient than other algorithms in achieving that performance.…”

Section: Setupsmentioning

confidence: 99%

“…The average bias lets us know whether Q θ is overestimated or underestimated, while standard deviation measures whether Q θ is overfitting. Sample efficiency (SE) (Chen et al, 2021;Dorner, 2021) is measured by the ratio of the number of samples collected when RAC and some algorithms reach the specified performance. Hopper is not in the comparison object as the performance of algorithms is almost indistinguishable.…”

Section: Setupsmentioning

confidence: 99%

See 2 more Smart Citations

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

Tang

Pang

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

IntroductionThe value approximation bias is known to lead to suboptimal policies or catastrophic overestimation bias accumulation that prevent the agent from making the right decisions between exploration and exploitation. Algorithms have been proposed to mitigate the above contradiction. However, we still lack an understanding of how the value bias impact performance and a method for efficient exploration while keeping stable updates. This study aims to clarify the effect of the value bias and improve the reinforcement learning algorithms to enhance sample efficiency.MethodsThis study designs a simple episodic tabular MDP to research value underestimation and overestimation in actor-critic methods. This study proposes a unified framework called Realistic Actor-Critic (RAC), which employs Universal Value Function Approximators (UVFA) to simultaneously learn policies with different value confidence-bound with the same neural network, each with a different under overestimation trade-off.ResultsThis study highlights that agents could over-explore low-value states due to inflexible under-overestimation trade-off in the fixed hyperparameters setting, which is a particular form of the exploration-exploitation dilemma. And RAC performs directed exploration without over-exploration using the upper bounds while still avoiding overestimation using the lower bounds. Through carefully designed experiments, this study empirically verifies that RAC achieves 10x sample efficiency and 25% performance improvement compared to Soft Actor-Critic in the most challenging Humanoid environment. All the source codes are available at https://github.com/ihuhuhu/RAC.DiscussionThis research not only provides valuable insights for research on the exploration-exploitation trade-off by studying the frequency of policies access to low-value states under different value confidence-bounds guidance, but also proposes a new unified framework that can be combined with current actor-critic methods to improve sample efficiency in the continuous control domain.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Setupsmentioning

confidence: 99%

Section: Setupsmentioning

confidence: 99%

See 1 more Smart Citation

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

Tang

Pang

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…As an example, the value approximation function was replaced in Ref. [82] with the Kanerva-based function approximation. Although performance improvement was evidenced, the algorithms were still not suitable for large scale networks.…”

Section: Multi-agent Reinforcement Learning Approach For Joint Sensin...mentioning

confidence: 99%

Towards a multi-agent reinforcement learning approach for joint sensing and sharing in cognitive radio networks

Rapetswa

Cheng

2023

Intell. and Converged Netw.

View full text Add to dashboard Cite

The adoption of the Fifth Generation (5G) and beyond 5G networks is driving the demand for learning approaches that enable users to co-exist harmoniously in a multi-user distributed environment. Although resourceconstrained, the Cognitive Radio (CR) has been identified as a key enabler of distributed 5G and beyond networks due to its cognitive abilities and ability to access idle spectrum opportunistically. Reinforcement learning is well suited to meet the demand for learning in 5G and beyond 5G networks because it does not require the learning agent to have prior information about the environment in which it operates. Intuitively, CRs should be enabled to implement reinforcement learning to efficiently gain opportunistic access to spectrum and co-exist with each other. However, the application of reinforcement learning is straightforward in a single-agent environment and complex and resource intensive in a multi-agent and multi-objective learning environment. In this paper, (1) we present a brief history and overview of reinforcement learning and its limitations; (2) we provide a review of recent multi-agent learning methods proposed and multi-agent learning algorithms applied in Cognitive Radio (CR) networks; and (3) we further present a novel framework for multi-CR reinforcement learning and conclude with a synopsis of future research directions and recommendations.

show abstract

“…We implement RAC with SAC and TD3 as RAC-SAC and RAC-TD3(see more details in Appendix B). We compare to state-of-the-art algorithms: SAC [17], TD3 [15], MBPO [20], REDQ [10] and [10,12] is measure by the ratio of the number of samples collected when RAC and some algorithm reach the specified performance. The last 4 rows show how many times REDQ are more sample efficient than other algorithms in reaching that performance.…”

Section: Setupsmentioning

confidence: 99%

Balancing Value Underestimation and Overestimation with Realistic Actor-Critic

Li¹,

Tang²,

Pang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Model-free deep reinforcement learning (RL) has been successfully applied to challenging continuous control domains. However, poor sample efficiency prevents these methods from being widely used in real-world domains. This paper introduces a novel model-free algorithm, Realistic Actor-Critic(RAC), which can be incorporated with any off-policy RL algorithms to improve sample efficiency. RAC employs Universal Value Function Approximators (UVFA) to simultaneously learn a policy family with the same neural network, each with different trade-offs between underestimation and overestimation. To learn such policies, we introduce uncertainty punished Q-learning, which uses uncertainty from the ensembling of multiple critics to build various confidence-bounds of Q-function. We evaluate RAC on the MuJoCo benchmark, achieving 10x sample efficiency and 25% performance improvement on the most challenging Humanoid environment compared to SAC.Preprint. Under review.

show abstract

Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Cited by 3 publications

References 66 publications

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

Realistic Actor-Critic: A framework for balance between value overestimation and underestimation

Towards a multi-agent reinforcement learning approach for joint sensing and sharing in cognitive radio networks

Balancing Value Underestimation and Overestimation with Realistic Actor-Critic

Contact Info

Product

Resources

About