A Rank-Based Sampling Framework For Offline Reinforcement Learning

Shen, Yichen; Fang, Zhou; Xu, Yunkun; Cao, Yu; Zhu, Jiangcheng

doi:10.1109/cei52496.2021.9574597

2021 IEEE International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology ( 2021

DOI: 10.1109/cei52496.2021.9574597

|View full text |Cite

A Rank-Based Sampling Framework For Offline Reinforcement Learning

Yichen Shen

Zhou Fang

Yunkun Xu

et al.

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2022

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, few works apply data rebalance to offline RL (not including imitation learning). One work is RBS(Shen et al, 2021), which also focuses on data perspective. However, RBS designs more sophisticated strategies like upper envelope to dynamically modify the sampling weights, while our method adopts more simple and efficient static rebalance before training.…”

mentioning

confidence: 99%

Boosting Offline Reinforcement Learning via Data Rebalancing

Yang¹,

Kang²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. To address this problem, existing works mainly focus on designing sophisticated algorithms to explicitly or implicitly constrain the learned policy to be close to the behavior policy. The constraint applies not only to well-performing actions but also to inferior ones, which limits the performance upper bound of the learned policy. Instead of aligning the densities of two distributions, aligning the supports gives a relaxed constraint while still being able to avoid out-of-distribution actions. Therefore, we propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. More specifically, we construct a better behavior policy by resampling each transition in an old dataset according to its episodic return. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time. Extensive experiments demonstrate that ReD is effective at boosting offline RL performance and orthogonal to decoupling strategies in long-tailed classification. New state-of-the-arts are achieved on the D4RL benchmark.

show abstract

mentioning

confidence: 99%

Boosting Offline Reinforcement Learning via Data Rebalancing

Yang¹,

Kang²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

hi@scite.ai

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

A Rank-Based Sampling Framework For Offline Reinforcement Learning

Cited by 1 publication

References 4 publications

Boosting Offline Reinforcement Learning via Data Rebalancing

Boosting Offline Reinforcement Learning via Data Rebalancing

Contact Info

Product

Resources

About