An adjustment method of the number of states on Q‐learning segmenting state space adaptively

Abstract

SUMMARYThe results of imposing limitations on the number of states and of promoting the splitting of states in Q-learning are presented. Q-learning is a common reinforcement learning method in which the learning agent autonomously segments the environment states. In situations where the designer of an agent is unable to explicitly provide the agent with the boundaries of states in the environment in which the agent is acting, the agent needs to simultaneously learn while autonomously determining the internal discrete states that are needed in order to take the appropriate actions. A simple method of segmenting states based on a reinforcement signal (QLASS) has been proposed for this purpose. However, the original method suffers from the problem that the number of states grows excessively large as learning proceeds. A method is therefore proposed that defines temperature and eligibility attributes for each of the internal discrete states of the agent, and that limits and adds to the number of internal discrete states, and promotes random actions depending on the values of these attributes. The results of applying the proposed method to a number of tasks, including tasks that incorporate a dynamic environment, are compared to the QLASS method when only the reinforcement signal is used, and a similar level of learning results is found to be achieved using a fewer number of states. Furthermore, it is found that tasks are able to be completed in a small number of steps even when only a small number of trials are used for learning.

Reinforcement learning based on modular fuzzy model with gating unit

“…To avoid the curse of dimensionality, there exists modular hierarchical learning [10,11,16] that construct the learning model as the combination of subspaces. Adaptive segmentation [12,13] for constructing the learning model validly corresponding to the environment is also studied. However more effective technique of different approach is also necessary in order to apply reinforcement learning to actual sized problems.…”

A study on multi-agent reinforcement learning problem based on hierarchical modular fuzzy model

“…To avoid the curse of dimensionality, there exists modular hierarchical learning [10,11] that construct the learning model as the combination of subspaces. Adaptive segmentation [12,13] for constructing the learning model validly corresponding to the environment is also studied. However more effective technique of different approach is also necessary in order to apply reinforcement learning to actual sized problems.…”

State space segmentation for acquisition of agent behavior

“…QLASS (Q-Learning with Adaptive State Segmentation) is one of on-line categorization methods [8,24]. QLASS categorizes percept vectors on the basis of a Voronoi diagram, where each Voronoi cell corresponds to a category.…”

“…Since each category is treated as a state in re-inforcement learning, generating too many categories deteriorates the performance of reinforcement learning. In [8], Hamagami and his group proposed some heuristics to reduce the number of categories. In their method, however, the maximum number of categories must be specified although it is difficult to decide the optimal value of it.…”

“…SSED inherits some characteristics from QLASS [8] and fuzzy ART with category merging [35,36]. Similar to QLASS, SSED decides the nearest category to a current percept vector on the basis of Euclidean distance.…”