Learning Multirobot Hose Transportation and Deployment by Distributed Round-Robin Q-Learning

Abstract

Multi-Agent Reinforcement Learning (MARL) algorithms face two main difficulties: the curse of dimensionality, and environment non-stationarity due to the independent learning processes carried out by the agents concurrently. In this paper we formalize and prove the convergence of a Distributed Round Robin Q-learning (D-RR-QL) algorithm for cooperative systems. The computational complexity of this algorithm increases linearly with the number of agents. Moreover, it eliminates environment non sta tionarity by carrying a round-robin scheduling of the action selection and execution. That this learning scheme allows the implementation of Modular State-Action Vetoes (MSAV) in cooperative multi-agent systems, which speeds up learning convergence in over-constrained systems by vetoing state-action pairs which lead to undesired termination states (UTS) in the relevant state-action subspace. Each agent’s local state-action value function learning is an independent process, including the MSAV policies. Coordination of locally optimal policies to obtain the global optimal joint policy is achieved by a greedy selection procedure using message passing. We show that D-RR-QL improves over state-of-the-art approaches, such as Distributed Q-Learning, Team Q-Learning and Coordinated Reinforcement Learning in a paradigmatic Linked Multi-Component Robotic System (L-MCRS) control problem: the hose transportation task. L-MCRS are over-constrained systems with many UTS induced by the interaction of the passive linking element and the active mobile robots.

Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications

“…Regarding applications of multi-agent learning, there have been many studies using traditional MARL methods to solve various problems such as controlling a group of autonomous vehicles or drones [43], robot soccer [102], controlling traffic signals [75], coordinating collaborative bots in factories and warehouse [47], controlling electrical power networks [93] or optimizing distributed sensor networks [37], automated trading [88], machine bidding in competitive e-commerce and financial markets [9], resource management [44], transportation [21], and phenomena of social sciences [62]. Since the emergence of DQN [73], efforts to extend traditional RL to deep RL in the multi-agent domain have been found in the literature but they are still very limited (see Table 4 for applications available in the current literature).…”

Decentralized Control of Multi-Robot System in Cooperative Object Transportation Using Deep Reinforcement Learning

“…However, this research relied on a path planning algorithm and the function of the DQN algorithm was only to adjust the robots whenever a pre-planned trajectory is not accessible. In [29], a multi-agent reinforcement learning algorithm was proposed to deal with a hose transportation problem. The proposed algorithm was based on the original Q-learning, thus was not capable of continuous state inputs.…”

Decentralized Opportunistic Spectrum Resources Access Model and Algorithm toward Cooperative Ad-Hoc Networks

“…Many decentralized methods have referenced the design of POMDP, varying reliance on schemes and can only handle intermittent communication resource scheduling. Reinforcement learning (RL) [ 15 ] is a paradigm to solve POMDP problems, and it is inspired by a learning theory which has good performance in multi-robots decision applications [ 16 , 17 ]. For most RL-based multi-agent systems, the rewards are more achieved by long-team learning, which is the expected accumulated reward that the agent expects to receive in the future under the policy, and can be specified by update value function.…”