Multi-Robot Collaborative Perception With Graph Neural Networks

Abstract

Multi-robot systems such as swarms of aerial robots are naturally suited to offer additional flexibility, resilience, and robustness in several tasks compared to a single robot by enabling cooperation among the agents. To enhance the autonomous robot decision-making process and situational awareness, multi-robot systems have to coordinate their perception capabilities to collect, share, and fuse environment information among the agents efficiently to obtain context-appropriate information or gain resilience to sensor noise or failures. In this paper, we propose a general-purpose Graph Neural Network (GNN) with the main goal to increase, in multi-robot perception tasks, single robots' inference perception accuracy as well as resilience to sensor failures and disturbances. We show that the proposed framework can address multi-view visual perception problems such as monocular depth estimation and semantic segmentation. Several experiments both using photo-realistic and real data gathered from multiple aerial robots' viewpoints show the effectiveness of the proposed approach in challenging inference conditions including images corrupted by heavy noise and camera occlusions or failures.

Graph Neural Networks in IoT: A Survey

“…Combined with the graph structured relational information, detailed agents' states are fed into GNNs to generate final representations by aggregating neighborhood information. Various established works [36,41,59,66,115,130,138,146,148,159,180,189,190,219,260,280,330,331] demonstrate the superior performance of using GNNs in manifold research domains that involve multi-agent interactions. Table 1 summarizes the sensor infrastructures, GNN models, and learning targets in the collected works.…”

“…The most popular node feature extraction method is using Neural Networks (e.g., Multi-layer Perceptron (MLP), Recurrent Neural Networks) to encode descriptive node features and generate high-level embeddings of agents' states. As demonstrated in these works [36,59,115,130,138,146,152,159,180,189,280,330,331], researchers use MLPs and/or Long-short Term Memory (LSTM) neural networks to encode motion and location information of agents and then produce feature representations that capture spatial and temporal variations as the node attributes. Edge connection: edge information defines how the agents interact with each other in their situated environments.…”

“…The edges can indicate the influence between agents, where the movement of one agent can impact the movement of other agents. In the surveyed papers, the authors of the works [36,41,66,115,143,146,189,260,330,331] construct edges between two agents if and only if these two agents are close to each other, since the authors assume that interactions/communications can only be effective if two agents are within a bounded distance. Adjacency matrices are used to represent the edge connections within multi-agent environments, where 1 indicates adjacency and 0 indicates nonadjacency.…”

“…Particularly, the kernels of GNNs play an essential role in encoding the graph structured data and generating comprehensive embeddings that capture the complex interactions between agents by aggregating neighborhood information. As illustrated in Table 1, in general, there are three types of GNN kernels that have been adopted in modeling MAI: 1) by relying on message passing mechanisms, the authors of the works [36,41,138,159,219] apply MPNN/GN to model the interactions between agents; 2) based on graph convolution algorithms, the authors of the works [43,115,148,190,260,280,330] employ graph convolutions to generate high level representations of multi-agent interactions; 3) by leveraging attention mechanisms, the authors of the works [59,66,130,146,180,189,331] make the neural information propagation among neighborhood with selective focus. MPNN/GN-based GNNs: in this type of GNNs, given an input of graph structured data of MAI and agent states, MPNNs/GNs activate a message passing algorithm with finite-steps traveling through the graphs to update agent states.…”

Mean Square Consensus of General Linear Multiagent Systems with Communication Noises under Switching Topologies

“…In system and control community, the coordination problem of MASs is one of the most concerned hotspots in the past decade, which has shown its potential in real-world applications, such as distributed sensor networks, smart grids, and multirobot formation [1][2][3][4]. Consensus is a fundamental issue in the control problem of MASs, which refers to designing a protocol such that all agents converge to a common value.…”