Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining

Liu, Yueh-Cheng; Huang, Yu‐Kai; Chiang, Hung-Yueh; Su, Hung-Ting; Liu, Zheyu; Chen, Chin-Tang; Tseng, Ching-Yu; Hsu, Winston H.

doi:10.48550/arxiv.2104.04687

Cited by 8 publications

(25 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there is no intra-modal constraint on point clouds, which can lead to trivial solutions and limit performance improvement. On the contrary, (Liu et al 2021) propose a method to imbue the image prior to the 3D representation. All these methods motivate us to further explore multimodal pre-training strategies for more challenging outdoor multi-modal data.…”

Section: Multi-modal Representation Learningmentioning

confidence: 99%

“…Compared with methods that focus on the indoor RGB-D data (Hou et al 2021b;Liu et al 2021), which utilize a U-Net (Ronneberger, Fischer, and Brox 2015) shape backbone to align the dense/local feature extracted by point-cloud feature extractor, our method only adopts a standard ResNet-50 encoder to extract the global features. Since the modules apply to global representations, alignment is achieved on the fly.…”

Section: Inter-modal Feature Interaction Modulementioning

confidence: 99%

See 1 more Smart Citation

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

Chen

Li³

et al. 2022

AAAI

View full text Add to dashboard Cite

Pre-training has become a standard paradigm in many computer vision tasks. However, most of the methods are generally designed on the RGB image domain. Due to the discrepancy between the two-dimensional image plane and the three-dimensional space, such pre-trained models fail to perceive spatial information and serve as sub-optimal solutions for 3D-related tasks. To bridge this gap, we aim to learn a spatial-aware visual representation that can describe the three-dimensional space and is more suitable and effective for these tasks. To leverage point clouds, which are much more superior in providing spatial information compared to images, we propose a simple yet effective 2D Image and 3D Point cloud Unsupervised pre-training strategy, called SimIPU. Specifically, we develop a multi-modal contrastive learning framework that consists of an intra-modal spatial perception module to learn a spatial-aware representation from point clouds and an inter-modal feature interaction module to transfer the capability of perceiving spatial information from the point cloud encoder to the image encoder, respectively. Positive pairs for contrastive losses are established by the matching algorithm and the projection matrix. The whole framework is trained in an unsupervised end-to-end fashion. To the best of our knowledge, this is the first study to explore contrastive learning pre-training strategies for outdoor multi-modal datasets, containing paired camera images and LIDAR point clouds.

show abstract

Section: Multi-modal Representation Learningmentioning

confidence: 99%

Section: Inter-modal Feature Interaction Modulementioning

confidence: 99%

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

Chen

Li³

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…On a related note, for video as a 2D+1D data, TimeSFormer [4] proposes an inflated design from 2D transformers, plus memorizing information across frames using another transformer along the additional time dimension. [24] provides a pixel-topoint knowledge distillation by contrastive learning. It is also possible to apply a uniform transformer backbone in different data modalities, including 2D and 3D images, which is successfully shown by Preceiver [19] and Omnivore [15].…”

Section: Transferring Knowledge Between 2d and 3dmentioning

confidence: 99%

Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?

Wang¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision Transformers (ViTs) have proven to be effective, in solving 2D image understanding tasks by training over large-scale image datasets; and meanwhile as a somehow separate track, in modeling the 3D visual world too such as voxels or point clouds. However, with the growing hope that transformers can become the "universal" modeling tool for heterogeneous data, ViTs for 2D and 3D tasks have so far adopted vastly different architecture designs that are hardly transferable. That invites an (over-)ambitious question: can we close the gap between the 2D and 3D ViT architectures? As a piloting study, this paper demonstrates the appealing promise to understand the 3D visual world, using a standard 2D ViT architecture, with only minimal customization at the input and output levels without redesigning the pipeline. To build a 3D ViT from its 2D sibling, we "inflate" the patch embedding and token sequence, accompanied with new positional encoding mechanisms designed to match the 3D data geometry. The resultant "minimalist" 3D ViT, named Simple3D-Former, performs surprisingly robustly on popular 3D tasks such as object classification, point cloud segmentation and indoor scene detection, compared to highly customized 3D-specific designs. It can hence act as a strong baseline for new 3D ViTs. Moreover, we note that pursing a unified 2D-3D ViT design has practical relevance besides just scientific curiosity. Specifically, we demonstrate that Simple3D-Former naturally enables to exploit the wealth of pre-trained weights from large-scale realistic 2D images (e.g., ImageNet), which can be plugged in to enhancing the 3D task performance "for free". Our code is available at https://github.com/VITA-Group/Simple3D-Former.git.Preprint. Under review.

show abstract

“…For example, xMUDA [58] utilizes aligned images and point-clouds to transfer 2D feature map information for 3D semantic segmentation through knowledge distillation [59]. For cross-modal transfer learning [60], Liu et al [61] proposed pixel-topoint knowledge transfer (PPKT) from 2D to 3D which uses aligned RGB and RGB-D images during pretraining. Our work does not rely on joint image-point-cloud pretraining.…”

Section: Cross-modal Learningmentioning

confidence: 99%

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

Xu¹,

Yang²,

Galanti³

et al. 2021

Preprint

View full text Add to dashboard Cite

3D point-clouds and 2D images are different visual representations of the physical world. While human vision can understand both representations, computer vision models designed for 2D image and 3D point-cloud understanding are quite different. Our paper investigates the potential for transferability between these two representations by empirically investigating whether this approach works, what factors affect the transfer performance, and how to make it work even better. We discovered that we can indeed use the same neural net model architectures to understand both images and point-clouds. Moreover, we can transfer pretrained weights from image models to point-cloud models with minimal effort. Specifically, based on a 2D ConvNet pretrained on an image dataset, we can transfer the image model to a point-cloud model by inflating 2D convolutional filters to 3D then finetuning its input, output, and optionally normalization layers. The transferred model can achieve competitive performance on 3D point-cloud classification, indoor and driving scene segmentation, even beating a wide range of point-cloud models that adopt task-specific architectures and use a variety of tricks. The code is avaliable at: https://github.com/chenfengxu714/image2point.

show abstract

Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining

Cited by 8 publications

References 63 publications

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?

Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models

Contact Info

Product

Resources

About