Meta-RangeSeg: LiDAR Sequence Semantic Segmentation Using Multiple Feature Aggregation

Abstract

LiDAR sensor is essential to the perception system in autonomous vehicles and intelligent robots. To fulfill the realtime requirements in real-world applications, it is necessary to efficiently segment the LiDAR scans. Most of previous approaches directly project 3D point cloud onto the 2D spherical range image so that they can make use of the efficient 2D convolutional operations for image segmentation. Although having achieved the encouraging results, the neighborhood information is not wellpreserved in the spherical projection. Moreover, the temporal information is not taken into consideration in the single scan segmentation task. To tackle these problems, we propose a novel approach to semantic segmentation for LiDAR sequences named Meta-RangeSeg, where a novel range residual image representation is introduced to capture the spatial-temporal information. Specifically, Meta-Kernel is employed to extract the meta features, which reduces the inconsistency between the 2D range image coordinates input and Cartesian coordinates output. An efficient U-Net backbone is used to obtain the multi-scale features. Furthermore, Feature Aggregation Module (FAM) aggregates the meta features and multi-scale features, which tends to strengthen the role of range channel. We have conducted extensive experiments for performance evaluation on SemanticKITTI, which is the de-facto dataset for LiDAR semantic segmentation. The promising results show that our proposed Meta-RangeSeg method is more efficient and effective than the existing approaches. Our full implementation is publicly available at https://github.com/songw-zju/Meta-RangeSeg.

A comprehensive overview of deep learning techniques for 3D point cloud classification and semantic segmentation

“…In contrast to previous methods, this approach maintains neighborhood information more effectively and considers temporal information in single scan segmentation tasks. To address these issues, Wang et al [240] presented Meta-RangeSeg, which adopts a unique range residual image representation to collect spatial-temporal information. To capture the meta features, Meta-Kernel is used, which minimizes the discrepancy between the 2D range image coordinates input and the Cartesian coordinates output.…”

Deep Learning for 3D Reconstruction, Augmentation, and Registration: A Review Paper

“…We discussed in detail some of the most advanced and/or benchmarking deep learning methods for 3D object recognition in our earlier work. These methods covered a range of 3D data formats, such as RGB-D (IMVoteNet) [ 4 ], voxels (VoxelNet) [ 5 ], point clouds (PointRCNN) [ 6 ], mesh (MeshCNN) [ 7 ], and 3D video (Meta-RangeSeg) [ 1 , 8 ].…”

Exploring metadiscourse in university lectures in Hong Kong

“…Multi-scan Outdoor 3D Semantic Segmentation: Compared to single-scan semantic segmentation, the multi-scan task needs to discriminate the moving and stationary states of the objects based on temporal information. In addition to the simple fusion strategy discussed above, another stream of approaches [7,23,24,28] attempts to process each point cloud in a sequence separately and fuse the feature representations for temporal modeling. For instance, SpSe-quenceNet [24] proposes a U-Net-based architecture to extract per-frame features and combine features of two consecutive frames to gather temporal information.…”

“…As shown in Figure 1, the simple point cloud fusion strategy cannot effectively enable the model to distinguish the motion states of cars even with a state-of-the-art backbone network SPVCNN [25]. Recently, there have been some early attempts [7,23,24,28] to employ attention modules [24] and recurrent networks [7,23,28] to fuse information across different temporal frames. However, these approaches do not perform well on the multi-scan task due to the insufficiency of temporal representations and the limited feature extraction ability of the model.…”