Context Transfer in Reinforcement Learning Using Action-Value Functions

Mousavi, Amin; Araabi, Babak Nadjar; Ahmadabadi, Majid Nili

doi:10.1155/2014/428567

Cited by 7 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Taylor and Stone (2007) discusses the representation transfer problems in which agents have different representations. Mousavi et al (2014) considers the context transfer problem in which agents have the same environment with the same reward function but different contexts (i.e., different state or action space). Recent works mainly focus on the empirical applications, including Mousavi et al (2014); Ammar et al (2014);Parisotto et al (2016); Gupta et al (2017); Barreto et al (2017) and more.…”

Section: Related Workmentioning

confidence: 99%

“…Mousavi et al (2014) considers the context transfer problem in which agents have the same environment with the same reward function but different contexts (i.e., different state or action space). Recent works mainly focus on the empirical applications, including Mousavi et al (2014); Ammar et al (2014);Parisotto et al (2016); Gupta et al (2017); Barreto et al (2017) and more. Theoretical guarantees are lacking even in linear value function settings.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Transferred Q-learning

Chen¹,

Jordan²,

Li³

2022

Preprint

View full text Add to dashboard Cite

We consider Q-learning with knowledge transfer, using samples from a target reinforcement learning (RL) task as well as source samples from different but related RL tasks. We propose transfer learning algorithms for both batch and online Q-learning with offline source studies. The proposed transferred Q-learning algorithm contains a novel re-targeting step which enables vertical information-cascading along multiple steps in an RL task, besides the usual horizontal information-gathering as transfer learning (TL) for supervised learning. We establish first theoretical justifications of TL in RL tasks by showing a faster rate of convergence of the Q function estimation in the offline RL transfer, and a lower regret bound in the offline-to-online RL transfer under certain similarity assumptions. Empirical evidences from both synthetic and real datasets are presented to backup the proposed algorithm and our theoretical results.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Transferred Q-learning

Chen¹,

Jordan²,

Li³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…El-Laithy and Bogdan [ 28 ] presented a reinforcement learning framework for spiking networks with dynamic synapses. Mousavi et al [ 29 ] discussed the notion of context transfer in reinforcement learning tasks. However, few researchers apply reinforcement learning in text processing tasks.…”

Section: Related Workmentioning

confidence: 99%

Joint Extraction of Entities and Relations Using Reinforcement Learning and Deep Learning

Feng

Zhang

Hao

et al. 2017

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

We use both reinforcement learning and deep learning to simultaneously extract entities and relations from unstructured texts. For reinforcement learning, we model the task as a two-step decision process. Deep learning is used to automatically capture the most important information from unstructured texts, which represent the state in the decision process. By designing the reward function per step, our proposed method can pass the information of entity extraction to relation extraction and obtain feedback in order to extract entities and relations simultaneously. Firstly, we use bidirectional LSTM to model the context information, which realizes preliminary entity extraction. On the basis of the extraction results, attention based method can represent the sentences that include target entity pair to generate the initial state in the decision process. Then we use Tree-LSTM to represent relation mentions to generate the transition state in the decision process. Finally, we employ Q-Learning algorithm to get control policy π in the two-step decision process. Experiments on ACE2005 demonstrate that our method attains better performance than the state-of-the-art method and gets a 2.4% increase in recall-score.

show abstract

“…The first category of the methods generalizes experiences across multiple learning agents. In multi-agent learning [4]- [6], the agents exchange their expertise and knowledge to speed up individual learning. These methods are useful when the agents have different expertise.…”

Section: Introductionmentioning

confidence: 99%

“…The fourth category of methods looks into the normalized value space, called functional space, for similarity and gener-alization [6], [9]. The main idea in employing the functional space is that, for a pair of states, similarity in the normalized values of some more experienced actions increases the probability of similarity in the Q-values of less experienced ones.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Generalization in the Subspaces for Faster Model-Based Reinforcement Learning

Hashemzadeh

Hosseini

Ahmadabadi

2019

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Due to the lack of enough generalization in the statespace, common methods in Reinforcement Learning (RL) suffer from slow learning speed especially in the early learning trials. This paper introduces a model-based method in discrete statespaces for increasing the learning speed in terms of required experience (but not required computational time) by exploiting generalization in the experiences of the subspaces. A subspace is formed by choosing a subset of features in the original state representation (full-space). Generalization and faster learning in a subspace are due to many-to-one mapping of experiences from the full-space to each state in the subspace. Nevertheless, due to inherent perceptual aliasing in the subspaces, the policy suggested by each subspace does not generally converge to the optimal policy. Our approach, called Model Based Learning with Subspaces (MoBLeS), calculates confidence intervals of the estimated Q-values in the full-space and in the subspaces. These confidence intervals are used in the decision making, such that the agent benefits the most from the possible generalization while avoiding from detriment of the perceptual aliasing in the subspaces. Convergence of MoBLeS to the optimal policy is theoretically investigated. Additionally, we show through several experiments that MoBLeS improves the learning speed in the early trials.Index Terms-reinforcement learning, generalization in subspaces, learning speed, curse of dimensionality, value interval estimation arXiv:1710.08012v2 [stat.ML]

show abstract

Context Transfer in Reinforcement Learning Using Action-Value Functions

Cited by 7 publications

References 9 publications

Transferred Q-learning

Transferred Q-learning

Joint Extraction of Entities and Relations Using Reinforcement Learning and Deep Learning

Exploiting Generalization in the Subspaces for Faster Model-Based Reinforcement Learning

Contact Info

Product

Resources

About