On The Transferability of Deep-Q Networks

Abstract

Transfer Learning (TL) is an efficient machine learning paradigm that allows overcoming some of the hurdles that characterize the successful training of deep neural networks, ranging from long training times to the needs of large datasets. While exploiting TL is a well established and successful training practice in Supervised Learning (SL), its applicability in Deep Reinforcement Learning (DRL) is rarer.In this paper, we study the level of transferability of three different variants of Deep-Q Networks on popular DRL benchmarks as well as on a set of novel, carefully designed control tasks. Our results show that transferring neural networks in a DRL context can be particularly challenging and is a process which in most cases results in negative transfer. In the attempt of understanding why Deep-Q Networks transfer so poorly, we gain novel insights into the training dynamics that characterizes this family of algorithms.

Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning☆

“…Actor-critic methods [69,94] are hybrid approaches that use policy-based methods to improve a policy while also evaluating it by estimating its corresponding value function. Several studies, including [22,95], investigated the adaptability of value-based algorithms to environmental changes. They trained a value function on a source task and then transferred the value function's parameters to a new task that differed from the source task in transition dynamics.…”

“…MF methods enable quick and computationally efficient action selection at decision time. However, as recent research [22,23,24,25] has shown, the adaptability of MF frameworks to environmental changes does not appear to be promising. This is due to the fact that an MF agent cannot adapt cached values of all states to changes in the environment.…”