Self-Organizing Neural Networks Integrating Domain Knowledge and Reinforcement Learning

Abstract

The use of domain knowledge in learning systems is expected to improve learning efficiency and reduce model complexity. However, due to the incompatibility with knowledge structure of the learning systems and real-time exploratory nature of reinforcement learning (RL), domain knowledge cannot be inserted directly. In this paper, we show how self-organizing neural networks designed for online and incremental adaptation can integrate domain knowledge and RL. Specifically, symbol-based domain knowledge is translated into numeric patterns before inserting into the self-organizing neural networks. To ensure effective use of domain knowledge, we present an analysis of how the inserted knowledge is used by the self-organizing neural networks during RL. To this end, we propose a vigilance adaptation and greedy exploitation strategy to maximize exploitation of the inserted domain knowledge while retaining the plasticity of learning and using new knowledge. Our experimental results based on the pursuit-evasion and minefield navigation problem domains show that such self-organizing neural network can make effective use of domain knowledge to improve learning efficiency and reduce model complexity.

Latent Structure Matching for Knowledge Transfer in Reinforcement Learning

“…Carroll and Seppi [18] defined similarity in terms of tasks, and proposed several possible task similarity measures based on the transfer time, policy overlap, Q-values, and reward structure respectively to measure the similarity between tasks. Teng [19] proposed to use self-organization neural networks to make the effective use of domain knowledge to reduce model complexity in RL. Through minimizing the reconstruction error of a restricted Boltzmann machine simulating the behavioral dynamics of two compared Markov decision processes, Ammar and Eaton [20] gave a data-driven automated Markov decision process similarity metric.…”

“…Later, in order to solve the problems in the earlier works mentioned above, some automatic similarity estimation works [15][16][17][18][19][20][21] are presented. These works made use data-driven similarity metrics, including the Markov decision process (MDP) similarity metric [20], the Hausdorff metric [21], and the Kantorovich metric [21].…”

“…These works made use data-driven similarity metrics, including the Markov decision process (MDP) similarity metric [20], the Hausdorff metric [21], and the Kantorovich metric [21]. These works are highly interpretative, but computationally expensive and have trouble when handling situations involving a large number of tasks [19].…”

Fast Reinforcement Learning under Uncertainties with Self-Organizing Neural Networks

“…TD-FALCON has three modes of operation. In the PER-FORM mode, Algorithm 1 is used to select cognitive node J for deriving action choice a for state s. In the LEARN mode, TD-FALCON learns the effect of action choice a on state s. In the INSERT mode, domain knowledge can be assimilated into FALCON [16].…”

“…This means a suitable ρ c3 has to be used. Given that it is difficult to know a priori a suitable ρ c3 , ρ c3 was adapted iteratively [16] using…”