“…The major challenge in offline RL is distribution shift [20,37,39], where the learned policy might generate out-of-distribution actions, resulting in erroneous value backups. Prior offline RL methods address this issue by regularizing the learned policy to be "close" to the behavior policy [20,50,29,82,93,37,68,57], through variants of importance sampling [59,74,49,75,54], via uncertainty quantification on Q-values [2,37,82,43], by learning conservative Q-functions [39,36], and with model-based training with a penalty on out-ofdistribution states [34,91,53,4,76,60,42,92]. While current benchmarks in offline RL [19,25] contain datasets that involve multi-task structure, existing offline RL methods do not leverage the shared structure of multiple tasks and instead train each individual task from scratch.…”