Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning

Abstract

Early stopping based on the validation set performance is a popular approach to find the right balance between under-and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving. As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on underand overfitting detection on a small subset of the continuously collected experience not used for training. We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari 100k benchmark. The results demonstrate that one can better balance under-and overestimation by adjusting the UTD ratio with our approach compared to the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applications. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to a higher robustness with regard to other learning-related hyperparameters further reducing the amount of necessary tuning.

Data-Driven Energy Management of an Electric Vehicle Charging Station Using Deep Reinforcement Learning

“…This approach proves advantageous in handling large-scale problems where explicit enumeration becomes computationally infeasible. This enhances the suitability of DRL methods for real-world applications [28].…”

SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning

“…REDQ (Chen et al 2020) and its variant (Wu et al 2022b) employ ensembled value networks to mitigate the value bias caused by increased UTD. (Dorka, Welschehold, and Burgard 2023) addresses model overfitting by employing dynamic UTD. (Li et al 2022) investigates the factors contributing to inferior performance in high UTD learning.…”

Automating Control of Overestimation Bias for Reinforcement Learning

“…Adaptively Calibrated Critic (ACC) [29] is a concurrent method presenting essentially the same idea as this paper while using slightly different optimization procedure. The reason for a slight improvement of ACC over AdaTQC on Hopper is likely the limited range of possible values for η ∈ [0, 5].…”

“…Our paper organically continues this line of research on overestimation bias. One other work that is closely related to our paper is Adaptively Calibrated Critic (ACC) [29]. It is a concurrent workshop paper, that investigates basically the same idea, but only applied to TQC.…”