Who2com: Collaborative Perception via Learnable Handshake Communication

Abstract

In this paper, we propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task. Unlike existing work in robotics and multi-agent reinforcement learning, we formulate the problem as one where learned information must be shared across a set of agents in a bandwidth-sensitive manner to optimize for scene understanding tasks such as semantic segmentation. Inspired by networking communication protocols, we propose a multi-stage handshake communication mechanism where the neural network can learn to compress relevant information needed for each stage. Specifically, a target agent with degraded sensor data sends a compressed request, the other agents respond with matching scores, and the target agent determines who to connect with (i.e., receive information from). We additionally develop the AirSim-CP dataset and metrics based on the AirSim simulator where a group of aerial robots perceive diverse landscapes, such as roads, grasslands, buildings, etc. We show that for the semantic segmentation task, our handshake communication method significantly improves accuracy by approximately 20% over decentralized baselines, and is comparable to centralized ones using a quarter of the bandwidth.

Multi-Robot Collaborative Perception With Graph Neural Networks

“…Multi-robot perception also includes the problems related to communication and bandwidth limitation among the robot network. Who2com [16] proposes a multi-stage handshake communication mechanism that enables agents to share information with limited communication bandwidth. The degraded agent only connects with the selected agent to receive information.…”

“…It contains images from five robots flying in a group in the city environment and rotating along their own z-axis with different angular velocities. In the noisy version of the dataset, Airsim-MAP applies Gaussian blurring with random kernel size from 1 to 100 and Gaussian noise [16]. Each camera has over 50% probability of being corrupted and each frame has at least one noisy image in five views [17].…”

Multi-robot Coordination and Planning in Uncertain and Adversarial Environments

“…There is a growing trend of using learning-based methods, in particular graph neural networks (GNNs) for multirobot coordination [31,108]. These approaches have been shown to learn what, when, and who to communicate with depending on the task at hand, instead of following a rigid communication topology [109,110]. GNNs capture the interactions among robots by modeling their coordination as a graph where robots (nodes) share information with their neighbors through communion links (edges).…”

“…Even though GNN-based architecture only requires robots communicating with certain hop neighbors, some of these communications may not be necessary. Some studies have presented parsimonious communication strategies that are selective in when to communicate with neighbors [62,63,109] and which neighbors to communicate with [110] to cutoff unnecessary communications among neighbors. Built on this idea, the third future avenue can be embedding parsimonious communication protocols into GNN architectures for multi-robot coordination.…”

Learning Distilled Collaboration Graph for Multi-Agent Perception

“…Although it is bandwidth-efficient, each individual perception output could be noisy and incomplete, causing unsatisfying fusion results. To deal with the performance-bandwidth trade-off, intermediate collaboration [19,34,20] has been proposed to aggregate intermediate features across agents; see Fig. 1 (b).…”