VGAI: End-to-End Learning of Vision-Based Decentralized Controllers for Robot Swarms

Abstract

Decentralized coordination of a robot swarm requires addressing the tension between local perceptions and actions, and the accomplishment of a global objective. In this work, we propose to learn decentralized controllers based solely on raw visual inputs. For the first time, this integrates the learning of two key components: communication and visual perception, in one end-to-end framework. More specifically, we consider that each robot has access to a visual perception of the immediate surroundings, and communication capabilities to transmit and receive messages from other neighboring robots. Our proposed learning framework combines a convolutional neural network (CNN) for each robot to extract messages from the visual inputs, and a graph neural network (GNN) over the entire swarm to transmit, receive and process these messages in order to decide on actions. The use of a GNN and locally-run CNNs results naturally in a decentralized controller. We jointly train the CNNs and the GNN so that each robot learns to extract messages from the images that are adequate for the team as a whole. Our experiments demonstrate the proposed architecture in the problem of drone flocking and show its promising performance and scalability, e.g., achieving successful decentralized flocking for large-sized swarms.

A Comprehensive Review of Vision-Based Robotic Applications: Current State, Components, Approaches, Barriers, and Potential Solutions

“…System outperforms existing ones [82]; Vision-based control is a good alternative to model-based control [83]; Control protocol completes formation tasks with visibility constraints [84]; Method eliminates false targets and improves positioning precision [85]; System maintained an acceptable accuracy and stability [86]; A person successfully controlled the robotic arm using the system [87]; Framework shows the proposed robotic hardware's efficiency [88]; movement was feasible and convenient [89]. 2020 [90][91][92][93][94][95] Grasping under occlusion [90]; Recognition and manipulation of objects [91]; Controllers for decentralized robot swarms [92]; Robot manipulation via human demonstration [93]; Robot manipulator using Iris tracking [94]; Object tracking of visual servoing [95].…”

“…Robust grasping method for a robotic system [90]; Effective stereo algorithm for manipulation of objects [91]; Successful framework to control decentralized robot swarms [92]; Generalized framework for activity recognition from human demonstrations [93]; Real-time iris tracking method for the ophthalmic robotic system [94]; Successful method for conventional template matching [95].…”

“…Method's effectiveness validated through experiments [90]; R-CNN method is very stable [91]; Architecture shows promising performance for large-sized swarms [92]; Proposed approach achieves good generalized performance [93]; Tracker is suitable for the ophthalmic robotic system [94]; Control system demonstrates significant improvement to feature tracking and robot motion [95].…”

Graph Neural Networks in IoT: A Survey

“…Existing works assume that each robot has access to perceptions of the immediate surroundings and that it can communicate with neighbor robots. Therefore, the GNN applied here is to control the entire swarm to transmit, receive, and process these messages between neighbor robots in order to decide on actions [44,75,102,123,143,152,259,260,276]. Autonomous vehicles are a type of more advanced robots with rich sensors, and have the capability to drive autonomously and safely on the road.…”

Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks

“…This paper builds on and significantly extends our preliminary conference version of VGAI [18]. We have extended the GNN architecture to incorporate time memory by using recurrent graph neural networks.…”