Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision

Bjorck, Johan; Chen, Xiangyu; De Sa, Christopher; Gomes, Carla P.; Weinberger, Kilian Q.

doi:10.48550/arxiv.2102.13565

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such large-scale studies on network architecture in RL could be as impactful as theoretical innovations, and we encourage more focus on network architecture. Whereas larger architecture requires more compute resources, this can potentially be offset by asynchronous training or other methods designed to accelerate RL [8,16,17,44,54].…”

Section: Discussionmentioning

confidence: 99%

Towards Deeper Deep Reinforcement Learning with Spectral Normalization

Björck¹,

Gomes²,

Weinberger³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In computer vision and natural language processing, innovations in model architecture that lead to increases in model capacity have reliably translated into gains in performance. In stark contrast with this trend, state-of-the-art reinforcement learning (RL) algorithms often use only small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, focusing specifically on soft actor-critic (SAC) algorithms. We verify, empirically, that naïvely adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, we show that dataset size is not the limiting factor, and instead argue that intrinsic instability from the actor in SAC taking gradients through the critic is the culprit. We demonstrate that a simple smoothing method can mitigate this issue, which enables stable training with large modern architectures. After smoothing, larger models yield dramatic performance improvements for state-ofthe-art agents -suggesting that more "easy" gains may be had by focusing on model architectures in addition to algorithmic innovations.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%