Continuous state/action reinforcement learning: A growing self-organizing map approach

Montazeri, Hesam; Moradi, Sajjad; Safabakhsh, Reza

doi:10.1016/j.neucom.2010.11.012

Cited by 22 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these works assume that an effective policy for a particular target task is already accessible to the teacher, which is not the case in this work. SOM-based approaches have previously been used in RL for a number of applications such as improving learning speed (Tateyama, Kawata, & Oguchi, 2004) and representation in continuous state-action domains (Montazeri, Moradi, & Safabakhsh, 2011; Smith, 2002). In the context of scaling task knowledge for continual learning (Ring, 1994), Ring, Schaul, and Schmidhuber (2011) described a modular approach to assimilate the knowledge of complex tasks using a training process that closely resembles SOM.…”

Section: Related Workmentioning

confidence: 99%

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

Karimpanal

Bouffanais

2018

Adaptive Behavior

View full text Add to dashboard Cite

The idea of reusing or transferring information from previously learned tasks (source tasks) for the learning of new tasks (target tasks) has the potential to significantly improve the sample efficiency of a reinforcement learning agent. In this work, we describe a novel approach for reusing previously acquired knowledge by using it to guide the exploration of an agent while it learns new tasks. In order to do so, we employ a variant of the growing selforganizing map algorithm, which is trained using a measure of similarity that is defined directly in the space of the vectorized representations of the value functions. In addition to enabling transfer across tasks, the resulting map is simultaneously used to enable the efficient storage of previously acquired task knowledge in an adaptive and scalable manner. We empirically validate our approach in a simulated navigation environment, and also demonstrate its utility through simple experiments using a mobile micro-robotics platform. In addition, we demonstrate the scalability of this approach, and analytically examine its relation to the proposed network growth mechanism. Further, we briefly discuss some of the possible improvements and extensions to this approach, as well as its relevance to real world scenarios in the context of continual learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

Karimpanal

Bouffanais

2018

Adaptive Behavior

View full text Add to dashboard Cite

show abstract

“…In this task the agent has to solve a maze by reaching the goal state fixed at position (19,2). At each time step, the agent receives a reward of -1 until it reaches the goal state where it receives a reward of 0.…”

Section: Testbed Environmentsmentioning

confidence: 99%

“…Mountain car [12], robot navigation [13]- [15], puddle world [16], cart centering [17] and arm-control [18], [19] are examples of that nature. Basically most of these models are made by two Self Organizing Maps (SOM) networks; one is used to map the state space and the other is used for searching the action space [12], [18], [19].…”

Section: Introductionmentioning

confidence: 99%

A Temporal Difference GNG-Based Approach for the State Space Quantization in Reinforcement Learning Environments

Vieira

Adeodato

Gonçalvès

2013

2013 IEEE 25th International Conference on Tools With Artificial Intelligence

View full text Add to dashboard Cite

The main issue when using reinforcement learning algorithms is how the estimation of the value function can be mapped into states. In very few cases it is possible to use tables but in the majority of cases, the number of states either can be too large to be kept into computer memory or it is computationally too expensive to visit all states. State aggregation models like the self-organizing maps have been used to make this possible by generalizing the input space and mapping the value functions into the states. This paper proposes a new algorithm called TD-GNG that uses the Growing Neural Gas (GNG) network to solve reinforcement learning problems by providing a way to map value functions into states. In experimental comparison against TD-AVQ and uniform discretization in three reinforcement problems, the TD-GNG showed improvements in three aspects, namely, 1) reduction of the dimensionality of the problem, 2) increase the generalization and 3) reduction of the convergence time. Experiments have also show that TD-GNG found a solution using less memory than TD-AVQ and uniform discretization without loosing quality in the policy obtained.

show abstract

“…Unfortunately, this method cannot be adapted to non-stationary data sets. Hesam et al [18] suggested an algorithm that applies GSOM neural networks to reinforcement learning to achieve an optimal state-space representation through two growing Self-Organizing Maps. The significant advantage of this algorithm is its ability to process online data in real-time using adaptive mechanisms.…”

Section: Introductionmentioning

confidence: 99%

Biomimetic SLAM Algorithm Based on Growing Self-Organizing Map

2021

View full text Add to dashboard Cite

A Biomimetic SLAM Algorithm Based on Growing Self-Organizing Map (GSOM-BSLAM), inspired by spatial cognitive mechanism of mammalian hippocampus, is proposed to resolve uncertainty problems in location identification and lack of real-time performance in simultaneous localization and mapping. The algorithm connects activation characteristics of the place cell and neurons in the output layer of the neural network to construct a topological map of space using a self-organizing growable mapping neural network. It utilizes self-motion-aware information to obtain activation response of the place cell to estimate the robot position information, improving the localization accuracy and real-time performance of the system. Meanwhile, an accurate environmental cognitive map is finally created by incorporating colordepth images for closed-loop detection and error correction for spatial cell path integration. The proposed algorithm is validated using publicly available KITTI and St. Lucia datasets. The experimental results demonstrate that the proposed algorithm outperforms RatSALM by 37.8% and 36.5% in terms of localization accuracy and real-time performance, respectively, indicating good mapping capabilities.

show abstract

Continuous state/action reinforcement learning: A growing self-organizing map approach

Cited by 22 publications

References 16 publications

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

Self-organizing maps for storage and transfer of knowledge in reinforcement learning

A Temporal Difference GNG-Based Approach for the State Space Quantization in Reinforcement Learning Environments

Biomimetic SLAM Algorithm Based on Growing Self-Organizing Map

Contact Info

Product

Resources

About