VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning

Che, Wang; Luo, Xin; Ross, Keith W.; Li, Dongsheng

doi:10.48550/arxiv.2202.10324

Cited by 3 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pretraining backbone vision components is common in RL setups, since it often yields a higher end performance [25], [17], [32], [37], [39]. We therefore pretrain the patch embedder in the same self-supervised fashion as [7].…”

Section: B Airloc Modelmentioning

confidence: 99%

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Pirinen¹,

Samuelsson²,

Backsund³

et al. 2023

Linköping Electronic Conference Proceedings

View full text Add to dashboard Cite

Climate-induced disasters are and will continue to be on the rise, and thus search-and-rescue (SAR) operations, where the task is to localize and assist one or several people who are missing, become increasingly relevant. In many cases the rough location may be known and a UAV can be deployed to explore a given, confined area to precisely localize the missing people. Due to time and battery constraints it is often critical that localization is performed as efficiently as possible. In this work we approach this type of problem by abstracting it as an aerial view goal localization task in a framework that emulates a SAR-like setup without requiring access to actual UAVs. In this framework, an agent operates on top of an aerial image (proxy for a search area) and is tasked with localizing a goal that is described in terms of visual cues. To further mimic the situation on an actual UAV, the agent is not able to observe the search area in its entirety, not even at low resolution, and thus it has to operate solely based on partial glimpses when navigating towards the goal. To tackle this task, we propose AiRLoc, a reinforcement learning (RL)-based model that decouples exploration (searching for distant goals) and exploitation (localizing nearby goals). Extensive evaluations show that AiRLoc outperforms heuristic search methods as well as alternative learnable approaches, and that it generalizes across datasets, e.g. to disaster-hit areas without seeing a single disaster scenario during training. We also conduct a proof-of-concept study which indicates that the learnable methods outperform humans on average. Code and models have been made publicly available at https://github.com/aleksispi/airloc.

show abstract

Section: B Airloc Modelmentioning

confidence: 99%

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Pirinen¹,

Samuelsson²,

Backsund³

et al. 2023

Linköping Electronic Conference Proceedings

View full text Add to dashboard Cite

show abstract

“…We instead use action-conditioned self-supervised learning to learn task-relevant representation. Several works pretrain policies on offline demonstrations [38,45]. However, the training data is from the policy learning environment and requires expert policy to collect trajectories, limiting the scalability and efficiency.…”

Section: Related Work 21 Image-based Policy Learningmentioning

confidence: 99%

Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Zhang¹,

Peng²,

Zhou³

2022

Preprint

View full text Add to dashboard Cite

Deep visuomotor policy learning achieves promising results in control tasks such as robotic manipulation and autonomous driving, where the action is generated from the visual input by the neural policy. However, it requires a huge number of online interactions with the training environment, which limits its real-world application. Compared to the popular unsupervised feature learning for visual recognition, feature pretraining for visuomotor control tasks is much less explored. In this work, we aim to pretrain policy representations for driving tasks using hours-long uncurated YouTube videos. A new contrastive policy pretraining method is developed to learn action-conditioned features from video frames with action pseudo labels. Experiments show that the resulting actionconditioned features bring substantial improvements to the downstream reinforcement learning and imitation learning tasks, outperforming the weights pretrained from previous unsupervised learning methods. Code and models will be made publicly available.

show abstract

“…Besides the joint learning framework used in CURL and SAC+AE, Shelhamer et al [67] investigate a pretraining framework to combine SSL with RL, and use self-supervised loss as an intrinsic reward to further boost performance during online learning. Recent works on policy learning (e.g., [56,65,74,80,86]) also take advantage of the self-supervised learning in a multi-step framework and show its great potential in solving challenging visual-based problems.…”

Section: Observation On Pretraining Frameworkmentioning

confidence: 99%

“…MVP [80] follows MAE [32] to train a visual encoder and even outperforms supervised pre-training. RRL [65] and VRL3 [74] also benefit from pre-training a deeper visual encoder on large datasets like ImageNet [16].…”

Section: Related Workmentioning

confidence: 99%

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?

Li¹,

Shang²,

Das³

et al. 2022

Preprint

View full text Add to dashboard Cite

We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels. We extend the contrastive reinforcement learning framework (e.g., CURL) that jointly optimizes SSL and RL losses and conduct an extensive amount of experiments with various self-supervised losses. Our observations suggest that the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentation when the same amount of data and augmentation is used. We further perform an evolutionary search to find the optimal combination of multiple self-supervised losses for RL, but find that even such a loss combination fails to meaningfully outperform the methods that only utilize carefully designed image augmentations. Often, the use of self-supervised losses under the existing framework lowered RL performances. We evaluate the approach in multiple different environments including a real-world robot environment, and confirm that no single self-supervised loss or image augmentation method can dominate all environments and that the current framework for joint optimization of SSL and RL is limited. Finally, we empirically investigate the pretraining framework for SSL + RL and the properties of representations learned with different approaches.Preprint. Under review.

show abstract

VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning

Cited by 3 publications

References 18 publications

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?

Contact Info

Product

Resources

About