Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

Huang, Siyuan; Xie, Yichen; Zhu, Song‐Chun; Zhu, Yixin

doi:10.1109/iccv48922.2021.00647

Cited by 129 publications

(70 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the same time, it still needs distribution estimation task and reconstruction task to obtain good performance. Inspired by self-supervised learning in 2D images [10,11,18], Huang et al [24] used the idea of contrastive learning to realize self-supervised point cloud representation learning. However, they only focus on the invariant representation between different perspectives, and do not consider the relationship between global shape features and local structure features, leading to limited generalization ability and difficulty in further performance improvement on downstream tasks.…”

Section: Related Workmentioning

confidence: 99%

“…ShapeNet consists of 57,448 synthetic objects from 55 categories. We follow the processing method of Huang et al [24], and each processed point cloud contains 2048 points. Unless otherwise specified, our models are pre-trained on ShapeNet.…”

Section: Datasetmentioning

confidence: 99%

“…S3DIS contains real scan point clouds of 272 rooms scanned in six different regions, and each point is labeled as one of 13 categories as semantic segmentation. We follow Huang et al [24] to divide each room into 1m×1m blocks and only use the XYZ coordinates of each point as the model input.…”

Section: Datasetmentioning

confidence: 99%

“…Unsupervised or self-supervised learning methods have shown their effectiveness in different fields [1,14,17,24,36]. In the field of point clouds, several recent works have achieved promising performance using autoencoders and generative models for unsupervised representation learning of point clouds [1,14,17], but they only focus on local structural details and cannot capture higher-level semantic features.…”

Section: Introductionmentioning

confidence: 99%

“…Inspired by self-supervised learning in 2D images [10,11,18], Huang ta al. [24] utilizes contrastive learning to realize self-supervised point cloud representation learning. However, they do not consider the relationship between local structures and the complete point clouds, leading to limited generalization ability and much difficulty in further performance improvement on downstream tasks with pre-trained models.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

Fu¹,

Gao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a simple and general framework for self-supervised point cloud representation learning. Human beings understand the 3D world by extracting two levels of information and establishing the relationship between them. One is the global shape of an object, and the other is the local structures of it. However, few existing studies in point cloud representation learning explored how to learn both global shapes and local-to-global relationships without a specified network architecture. Inspired by how human beings understand the world, we utilize knowledge distillation to learn both global shape information and the relationship between global shape and local structures. At the same time, we combine contrastive learning with knowledge distillation to make the teacher network be better updated. Our method achieves the state-of-theart performance on linear classification and multiple other downstream tasks. Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method. Our code will be released soon.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

Fu¹,

Gao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Unsupervised Pre-training for 3D Object Detection with Transformer

Sun¹,

Huang²,

Sun³

et al. 2022

Pattern Recognition and Computer Vision

View full text Add to dashboard Cite

LiDAL: Inter-frame Uncertainty Based Active Learning for 3D LiDAR Semantic Segmentation

Bai

Wang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We propose LiDAL, a novel active learning method for 3D Li-DAR semantic segmentation by exploiting inter-frame uncertainty among LiDAR frames. Our core idea is that a well-trained model should generate robust results irrespective of viewpoints for scene scanning and thus the inconsistencies in model predictions across frames provide a very reliable measure of uncertainty for active sample selection. To implement this uncertainty measure, we introduce new inter-frame divergence and entropy formulations, which serve as the metrics for active selection. Moreover, we demonstrate additional performance gains by predicting and incorporating pseudo-labels, which are also selected using the proposed inter-frame uncertainty measure. Experimental results validate the effectiveness of LiDAL: we achieve 95% of the performance of fully supervised learning with less than 5% of annotations on the SemanticKITTI and nuScenes datasets, outperforming state-of-the-art active learning methods. Code release: https://github.com/hzykent/LiDAL

show abstract

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds

Cited by 129 publications

References 36 publications

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

Unsupervised Pre-training for 3D Object Detection with Transformer

LiDAL: Inter-frame Uncertainty Based Active Learning for 3D LiDAR Semantic Segmentation

Contact Info

Product

Resources

About