Sparse multi-task reinforcement learning

Abstract

U n c o r r e c t e d A u t h o r P r o o fIntelligenza Artificiale xx (20xx) Abstract. In multi-task reinforcement learning (MTRL), the objective is to simultaneously learn multiple tasks and exploit their similarity to improve the performance w.r.t. single-task learning. In this paper we investigate the case when all the tasks can be accurately represented in a linear approximation space using the same small subset of the original (large) set of features. This is equivalent to assuming that the weight vectors of the task value functions are jointly sparse, i.e., the set of their non-zero components is small and it is shared across tasks. Building on existing results in multi-task regression, we develop two multi-task extensions of the fitted Q-iteration algorithm. While the first algorithm assumes that the tasks are jointly sparse in the given representation, the second one learns a transformation of the features in the attempt of finding a more sparse representation. For both algorithms we provide a sample complexity analysis and numerical simulations.

Scalable transfer learning in heterogeneous, dynamic environments

“…Calandriello et al [64] recently propose interesting extensions to the fitted Q-iteration algorithm [59] based on sparse linear regression models. These extensions could enable an agent to efficiently share knowledge between tasks in heterogeneous domains to do feature selection, and to learn Q-value functions.…”

SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards

“…Kolter et al also studied the problem called "Hierarchical Apprenticeship Learning" to learn bipedal locomotion [18]. There is also some work in utlizing multi-task learning for RL [2]. In HIRL, we explore how we can leverage demonstrations that are possibly spatially and temporally varying to infer such hierarchical structure.…”

“…With the additional features, the learned rewards will not only reflect the current state of the robot, but also the current active sub-task, and with the augmented states, any policy learning agent will be able to make use of these rewards. We call the entire framework Hierarchical Inverse Reinforcement Learning (HIRL), and it addresses two problems: (1) given a set of featurized demonstration trajectories, learn the locally linear sub-tasks, (2) use the learned sub-tasks to construct local rewards that respect the global sequential structure.…”

Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach

“…where Θ 2,0 = |{i ∈ [m 1 ] | j∈[m 2 ] Θ 2 ij = 0}| is the number of non-zero rows of Θ. Such an optimization problem arises in a number of applications including sparse singular value decomposition and principal component analysis [117,82,58], sparse reduced-rank regression [20,83,35,36,112], and reinforcement learning [26,105,75,120,102]. Rather than considering convex relaxations of the optimization problem (1), we directly work with a nonconvex formulation.…”

“…Our proposed algorithm can be applied to the regression step of any MTRL algorithm (we chose Fitted Q-iteration (FQI) for presentation purposes) to solve for the optimal policies for MDPs. Compared to [26] which uses convex relaxation, our algorithm is much more efficient in high dimensions.…”