Latency-Aware Collaborative Perception

Abstract

Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which actively adapts asynchronous perceptual features from multiple agents to the same time stamp, promoting the robustness and effectiveness of collaboration. To achieve such a featurelevel synchronization, we propose a novel latency compensation module, called SyncNet, which leverages feature-attention symbiotic estimation and time modulation techniques. Experiments results show that the proposed latency aware collaborative perception system with SyncNet can outperforms the state-of-the-art collaborative perception method by 15.6% in the communication latency scenario and keep collaborative perception being superior to single agent perception under severe latency.

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

“…In the past decade, the attention mechanism has played an increasingly important role in computer vision [13,57,69,70], which also inspires collaborative perception research. Since feature selection and relationship exploring are vital issues in intermediate collaborative perception, some works [21,23,29,71,72,73,74] leverage attention mechanisms to exploit more dynamic and robust collaborative perception strategies. Due to flexibility, attention-based design becomes the dominant strategy in intermediate collaboration.…”

“…Therefore, it is of great practical significance to ensure robustness and safety for collaboration perception. In this subsection, we focus on advanced research on these issues and summarize collaborative modules designed to address these issues [21,24,31,53,74,85,86].…”

“…2) Communication Issues: Since collaborative perception relies on communication networks, communication issues such as latency, interruption and information loss also affect the effectiveness of collaboration. In recent years, several efforts [21,24,74] have begun to explore solutions to these problems.…”

“…Furthermore, Lei et al [74] proposes the first latency-aware collaborative perception system, which realizes a featurelevel synchronization. Considering feature and attention are closely linked and influence each other, the core module SyncNet estimates the coupling of collaborative feature and attention simultaneously.…”

V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception

“…Sooner or later, these autonomous systems will be deployed on roads at scale, opening up opportunities for cooperation between them. Previous works in [4], [5], [6], [7], [8], [9], [10], [11], [12] have demonstrated that by leveraging the Vehicle-to-Everything (V2X) communication technology, AVs and infrastructure can perform cooperative perception by using the shared sensing information and thus significantly enhance the perception performance. Despite the remarkable improvement, these works evaluate the proposed systems on the dataset with natural scenarios that do not contain sufficient safety-critical scenes.…”