Collaborative Perception—The Missing Piece in Realizing Fully Autonomous Driving

Abstract

Environment perception plays a crucial role in enabling collaborative driving automation, which is considered to be the ground-breaking solution to tackling the safety, mobility, and sustainability challenges of contemporary transportation systems. Despite the fact that computer vision for object perception is undergoing an extraordinary evolution, single-vehicle systems’ constrained receptive fields and inherent physical occlusion make it difficult for state-of-the-art perception techniques to cope with complex real-world traffic settings. Collaborative perception (CP) based on various geographically separated perception nodes was developed to break the perception bottleneck for driving automation. CP leverages vehicle-to-vehicle and vehicle-to-infrastructure communication to enable vehicles and infrastructure to combine and share information to comprehend the surrounding environment beyond the line of sight and field of view to enhance perception accuracy, lower latency, and remove perception blind spots. In this article, we highlight the need for an evolved version of the collaborative perception that should address the challenges hindering the realization of level 5 AD use cases by comprehensively studying the transition from classical perception to collaborative perception. In particular, we discuss and review perception creation at two different levels: vehicle and infrastructure. Furthermore, we also study the communication technologies and three different collaborative perception message-sharing models, their comparison analyzing the trade-off between the accuracy of the transmitted data and the communication bandwidth used for data transmission, and the challenges therein. Finally, we discuss a range of crucial challenges and future directions of collaborative perception that need to be addressed before a higher level of autonomy hits the roads.

Multi-Layered Local Dynamic Map for a Connected and Automated In-Vehicle System

“…Significant contributions to CP research include the following: Seong-Woo Kim et al [16][17][18], who created a framework to extend perception beyond line-of-sight, a cooperative driving system using CP, and methods for improving AD safety and smoothness; Pierre Merdiganc et al [19], who integrated perception and vehicle-to-pedestrian communication to enhance Vulnerable Road Users' (VRUs) safety; Aaron Miller et al [20], who developed a perception and localization system allowing vehicles with basic sensors to leverage data from those with advanced sensors, thus elevating AD capabilities; Xiaboo Chen et al [21,22], who proposed a recursive Bayesian framework for more reliable cooperative tracking, and a robust framework for multi-vehicle tracking under inaccurate self-localization; Adamey et al [23], who introduced a method for collaborative vehicle tracking in mixed-traffic settings; Francesco Biral et al [24], who demonstrated how the SAFE STRIP EU project technology aids in deploying the LDM for Cooperative ITS safety applications; and Stefano Masi et al [25], who developed a cooperative roadside vision system to enhance the perception capabilities of an AV; Sumbal Malik et al [26], who highlight the need for advanced CP to overcome challenges in achieving level 5 AD; Tania Cerquitelli et al [27], who discussed in a special issue the integration of machine learning and artificial intelligence technologies to empower network communication, analysing how computer networks can become smarter; Andrea Piazzoni et al [28], who discuss how to model CP errors in AD, focusing on the impact of occlusion on safety and how CP may address it; Zhiying Song et al [29], who presented a framework for evaluating CP in connected AVs, emphasizing the importance of CP in increasing vehicle awareness beyond sensor FoV; Mao Shan et al [30], who introduced a novel framework for enhancing CP in Connected AVs by probabilistically fusing V2X data, improving perception range and decision-making in complex environments.…”