A multi-agent cooperative reinforcement learning model using a hierarchy of consultants, tutors and workers

Abstract

The hierarchical organisation of distributed systems can provide an efficient decomposition for machine learning. This paper proposes an algorithm for cooperative policy construction for independent learners, named Q-learning with aggregation (QA-learning). The algorithm is based on a distributed hierarchical learning model and utilises three specialisations of agents: workers, tutors and consultants. The consultant agent incorporates the entire system in its problem space, which it decomposes into sub-problems that are assigned to the tutor and worker agents. The QAlearning algorithm aggregates the Q-tables of worker agents into a central repository managed by their tutor agent. Each tutor's Q-table is then incorporated into the consultant's Qtable, resulting in a Q-table for the entire problem. The algorithm was tested using a distributed hunter prey problem, and experimental results show that QA-learning converges to a solution faster than single agent Q-learning and some famous cooperative Q-learning algorithms.

A Behavior Optimization Method for Unmanned Combat Aerial Vehicles Using Matrix Factorization

“…To the best of our knowledge, there is no RLbased method that attempts to infer the optimal behavior of UVs. Therefore, we builtR by using Q-learning algorithm estimating the cumulative reward of each situation-action pair in a Q-table, which is a basic method of RL [30]. Its performance largely depends on the environment design, and we utilized the same settings as that of AM and MF.…”

Bat Q-learning Algorithm

“…An MDP comprises a set of states } ,..., , { = indicates that the transition is invalid. The immediate expected reward for executing this transition is the deterministic reward ) , ( z x a s R [3]. It is important to note that the implementation of Q-learning to stochastic MDPs is beyond the scope of this paper.…”

“…As a consequence, AVE-Q may produce an incorrect policy, because it does not remove the bad Qvalues at the interaction stage [3].…”

“…Q-learning is a well known reinforcement learning (RL) algorithm that allows machines and software agents to develop an ideal behavior within a specific environment based on trial and error [1]- [3]. A Q-learning agent learns how to behave by trying actions to determine how to maximize some reward.…”

“…Such an approach to RL, which is called cooperative RL, is increasingly used by research labs around the world to solve real world problems, such as robot control and autonomous navigation [4], [5]. This is because cooperative reinforcement learners can learn and converge faster than independent reinforcement learners via sharing of information (e.g., Q-values, Episodes, Policies) [3], [6]- [8]. One such example is cooperative Q-learning, in which several learners share their Q-values among each other in order to accelerate their convergence to optimal solutions [9], [10].…”