Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration

Abstract

A major challenge in real-world reinforcement learning (RL) is the sparsity of reward feedback. Often, what is available is an intuitive but sparse reward function that only indicates whether the task is completed partially or fully. However, the lack of carefully designed, fine grain feedback implies that most existing RL algorithms fail to learn an acceptable policy in a reasonable time frame. This is because of the large number of exploration actions that the policy has to perform before it gets any useful feedback that it can learn from. In this work, we address this challenging problem by developing an algorithm that exploits the offline demonstration data generated by a sub-optimal behavior policy for faster and efficient online RL in such sparse reward settings. The proposed algorithm, which we call the Learning Online with Guidance Offline (LOGO) algorithm, merges a policy improvement step with an additional policy guidance step by using the offline demonstration data. The key idea is that by obtaining guidance from -not imitating -the offline data, LOGO orients its policy in the manner of the sub-optimal policy, while yet being able to learn beyond and approach optimality. We provide a theoretical analysis of our algorithm, and provide a lower bound on the performance improvement in each learning episode. We also extend our algorithm to the even more challenging incomplete observation setting, where the demonstration data contains only a censored version of the true state observation. We demonstrate the superior performance of our algorithm over state-of-the-art approaches on a number of benchmark environments with sparse rewards and censored state. Further, we demonstrate the value of our approach via implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance, where it shows excellent performance.

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

“…The idea here is to utilize expert's demonstrations to guide the standard learning procedure in RL algorithms [7], [8], [18], [34], [35]. Authors in [21], [36] proposed to include expert demonstrations to replay buffers and utilize them to accelerate the learning.…”

“…But as mentioned previously, the major drawback here is also their dependence upon the availability of demonstrations, which are hard to get in practice for continuous control problems. For instance, expert's demonstrations in [8] are obtained by running TRPO with dense rewards and then later used to train a policy with sparse rewards in the same environment. This could be difficult to achieve in practice.…”

“…Reward engineering for robotic tasks is difficult due to complex state space representations and usually requires manually-designed perception systems of the environment [5]. Hence, it makes sense to work directly with naturally specified sparse rewards [6]- [8]. For example, it is much easier to specify a binary reward (1 for successful completion of a task and 0 otherwise) than to come up with a dense reward structure.…”

“…For instance, authors in [19] induce such behaviors via intrinsic curiosity, and [20] utilizes information to motivate the exploration. Another line of work utilizes expert's demonstrations to learn effectively in sparse reward environments [7], [8], [14], [21]. The main idea here is to either use available demonstration to clone an expert's behavior (imitation learning) or just utilize demonstrations to provide additional rewards to guide the exploration [7], [8].…”

“…Another line of work utilizes expert's demonstrations to learn effectively in sparse reward environments [7], [8], [14], [21]. The main idea here is to either use available demonstration to clone an expert's behavior (imitation learning) or just utilize demonstrations to provide additional rewards to guide the exploration [7], [8]. But the major limitation of these approaches depends on the quality of the expert demonstrations.…”