LiCaNet: Further Enhancement of Joint Perception and Motion Prediction based on Multi-Modal Fusion

Abstract

<p>The safety and reliability of autonomous driving pivots on the accuracy of perception and motion prediction pipelines, which in turn reckons primarily on the sensors deployed onboard. Slight confusion in perception and motion prediction can result in catastrophic consequences due to misinterpretation in later pipelines. Therefore, researchers have recently devoted considerable effort towards developing accurate perception and motion prediction models. To that end, we propose LIDAR Camera network (LiCaNet) that leverages multi-modal fusion to further enhance the joint perception and motion prediction performance accomplished in our earlier work. LiCaNet expands on our previous fusion network by adding a camera image to the fusion of RV image with historical BEV data sourced from a LIDAR sensor. We present a comprehensive evaluation to validate the outstanding performance of LiCaNet compared to the state-of-the-art. Experiments reveal that utilizing a camera sensor results in a substantial perception gain over our previous fusion network and a steep reduction in displacement errors. Moreover, the majority of the achieved improvement falls within camera range, with the highest registered for small and distant objects, confirming the significance of incorporating a camera sensor into a fusion network.</p>

LiCaNext: Incorporating Sequential Range Residuals for Additional Advancement in Joint Perception and Motion Prediction

“…The two input LIDAR representations employed in [2] are BEV and RV images. Lastly, recently proposed LiCaNet [1] extends [2] with camera image fusion. LiCaNet records excellent performance for both perception and motion prediction compared to its predecessor.…”

“…Empty cells are filled with a value of −1. Further details on the projection algorithm can be found in [1].…”

“…They consist of spatio-temporal information sourced from two representations (BEV and RV), physical object dimensions encoded in the input BEV images, occlusion information provided from RV images, and rich semantics signified in a camera image. When these features are inserted into MotionNet backbone network, they yield accurate pixel-wise joint perception and motion predic- Output: [1], :] += 1 end P /= count // average (avoid division by 0) mask = (count == 0) P[mask, :] = -1 // assign -1 to empty cells end tion in real-time.…”

“…The first experiment records the performance of the original MotionNet model, which acts as the primary baseline. LiCaNet [1] proved that engaging RV and camera images into the fusion process outperform the baseline, which depends merely on BEV images as input. Our proposed LiCaNext expands on LiCaNet by incorporating residual images pushing the performance even further.…”

“…Following [1], the learning rate is initialized at 1.6×10 −3 and terminated at 0.8 × 10 −3 , with a decay factor of 0.5 every 10 epochs. The time span between consecutive LIDAR sweeps used to construct the historical BEVs and the sequential residual range images is 0.2s.…”