Active deep Q-learning with demonstration

Chen, Si-An; Tangkaratt, Voot; Lin, Hung‐Mo; Sugiyama, Masashi

doi:10.1007/s10994-019-05849-4

Cited by 17 publications

(15 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The learner may receive feedback in the form of sequences of actions planned by a teacher [10]. Uncertainty-based query was used in [14] but is limited to DQN [53], limiting the possible applications of this method. In contrast, our method can be combined with most of off-policy RL algorithms; and introduces the idea of goal-driven demonstrations.…”

Section: Learning From Interactive Human Feedbackmentioning

confidence: 99%

Goal-driven active learning

Bougie

Ichise

2021

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Deep reinforcement learning methods have achieved significant successes in complex decision-making problems. In fact, they traditionally rely on well-designed extrinsic rewards, which limits their applicability to many real-world tasks where rewards are naturally sparse. While cloning behaviors provided by an expert is a promising approach to the exploration problem, learning from a fixed set of demonstrations may be impracticable due to lack of state coverage or distribution mismatch—when the learner’s goal deviates from the demonstrated behaviors. Besides, we are interested in learning how to reach a wide range of goals from the same set of demonstrations. In this work we propose a novel goal-conditioned method that leverages very small sets of goal-driven demonstrations to massively accelerate the learning process. Crucially, we introduce the concept of active goal-driven demonstrations to query the demonstrator only in hard-to-learn and uncertain regions of the state space. We further present a strategy for prioritizing sampling of goals where the disagreement between the expert and the policy is maximized. We evaluate our method on a variety of benchmark environments from the Mujoco domain. Experimental results show that our method outperforms prior imitation learning approaches in most of the tasks in terms of exploration efficiency and average scores.

show abstract

Section: Learning From Interactive Human Feedbackmentioning

confidence: 99%

Goal-driven active learning

Bougie

Ichise

2021

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…Deep RL is a considerably new domain for the action advising studies. [19] introduced a novel LfD setup in which the demonstrations dataset is built interactively as in action advising. To do so, they employed uncertainty estimation capable models with LfD loss terms integrated in the learning stage.…”

Section: Related Workmentioning

confidence: 99%

“…In [11], an uncertainty-based advice collection strategy was proposed. According to this, the student adopts a multi-headed neural network architecture to access epistemic uncertainty estimations as in [19]. Later, [12] further studied the state novelty-based idea of [10] to devise a better studentinitiated advice collection method.…”

Section: Related Workmentioning

confidence: 99%

Learning on a Budget via Teacher Imitation

Ilhan¹,

Gow²,

Peŕez-Liebana³

2021

Preprint

View full text Add to dashboard Cite

Deep Reinforcement Learning (RL) techniques can benefit greatly from leveraging prior experience, which can be either self-generated or acquired from other entities. Action advising is a framework that provides a flexible way to transfer such knowledge in the form of actions between teacher-student peers. However, due to the realistic concerns, the number of these interactions is limited with a budget; therefore, it is crucial to perform these in the most appropriate moments. There have been several promising studies recently that address this problem setting especially from the student's perspective. Despite their success, they have some shortcomings when it comes to the practical applicability and integrity as an overall solution to the learning from advice challenge. In this paper, we extend the idea of advice reusing via teacher imitation to construct a unified approach that addresses both advice collection and advice utilisation problems. Furthermore, we also propose a method to automatically determine the relevant hyperparameters of these components on-the-fly to make it able to adapt to any task with minimal human intervention. The experiments we performed in 5 different Atari games verify that our algorithm can outperform its competitors by achieving state-of-the-art performance, and its components themselves also provides significant advantages individually.

show abstract

“…Based on deep Q-learning, researchers [26][27] proposed algorithms to help the agent formulate a more useful strategy when playing video games. Chen [28] proposed an algorithm that dynamically estimates the uncertainty of recent states, and utilizes the queried demonstration data by optimizing a supervised loss, in addition to the usual DQN loss [29]. Todd [30] presented an algorithm, Deep Q-learning from Demonstrations (DQfD), which leverages small sets of demonstration data to accelerate the learning process and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism.…”

Section: Related Workmentioning

confidence: 99%

Reversely Discovering and Modifying Properties Based on Active Deep Q-Learning

Huo

2020

IEEE Access

View full text Add to dashboard Cite

Many researchers studied DQN (Deep Q-Networks) to train a game AI to beat human players, while we trained an improved AI to reversely modify properties of 3D video games. Our ultimate objective is to improve automatic debug for software and cloud services. However, the problem that reversely discovers properties in online 3D Video Games in an automatic way has not been studied yet. Therefore, related special difficulties are first discussed in the paper. RMDQN (a Reverse Method based on our active Deep Q-Networks) is proposed to deal with the problem, and an active DQN is invented to make the reverse procedure automatic and intelligent. The action engine of RMDQN is able to control any operational game object like a player is playing, which makes automatic debug possible. A video demonstration is provided to show the result of reversely modifying game properties by our method. It was proved that our method can improve debug technology in 3D video games, and it will be applied in cloud services with few modifications.

show abstract

Active deep Q-learning with demonstration

Cited by 17 publications

References 9 publications

Goal-driven active learning

Goal-driven active learning

Learning on a Budget via Teacher Imitation

Reversely Discovering and Modifying Properties Based on Active Deep Q-Learning

Contact Info

Product

Resources

About