CoCoNets: Continuous Contrastive 3D Scene Representations

Lal, Shamit; Prabhudesai, Mihir; Mediratta, Ishita; Harley, Adam W.; Fragkiadaki, Katerina

doi:10.1109/cvpr46437.2021.01230

Cited by 10 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To make full use of the point cloud data, Contrast Context (Hou et al 2021a) proposes to adopt a ShapeContext descriptor to divide the scene, which provides more negative pairs for contrastive learning and improve the effectiveness of pre-training models. CoCoNets (Lal et al 2021) further explores self-supervised learning of amodal 3D feature representations agnostic to object and scene semantic content. The above methods focus on indoor RGB-D data.…”

Section: D Self-supervised Representation Learningmentioning

confidence: 99%

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

Chen

Li³

et al. 2022

AAAI

View full text Add to dashboard Cite

Pre-training has become a standard paradigm in many computer vision tasks. However, most of the methods are generally designed on the RGB image domain. Due to the discrepancy between the two-dimensional image plane and the three-dimensional space, such pre-trained models fail to perceive spatial information and serve as sub-optimal solutions for 3D-related tasks. To bridge this gap, we aim to learn a spatial-aware visual representation that can describe the three-dimensional space and is more suitable and effective for these tasks. To leverage point clouds, which are much more superior in providing spatial information compared to images, we propose a simple yet effective 2D Image and 3D Point cloud Unsupervised pre-training strategy, called SimIPU. Specifically, we develop a multi-modal contrastive learning framework that consists of an intra-modal spatial perception module to learn a spatial-aware representation from point clouds and an inter-modal feature interaction module to transfer the capability of perceiving spatial information from the point cloud encoder to the image encoder, respectively. Positive pairs for contrastive losses are established by the matching algorithm and the projection matrix. The whole framework is trained in an unsupervised end-to-end fashion. To the best of our knowledge, this is the first study to explore contrastive learning pre-training strategies for outdoor multi-modal datasets, containing paired camera images and LIDAR point clouds.

show abstract

Section: D Self-supervised Representation Learningmentioning

confidence: 99%

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

Chen

Li³

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Contrastive Learning for 3D. Contrastive learning [11,12,26], initially designed for 2D, was extended to 3D using similar points from different views in [27,36,81,91]. Limited training data is considered in [27].…”

Section: Related Workmentioning

confidence: 99%

COARSE3D: Class-Prototypes for Contrastive Learning in Weakly-Supervised 3D Point Cloud Segmentation

Li,

Cao,

de Charette

2022

Preprint

View full text Add to dashboard Cite

Annotation of large-scale 3D data is notoriously cumbersome and costly. As an alternative, weakly-supervised learning alleviates such a need by reducing the annotation by several order of magnitudes. We propose COARSE3D, a novel architecture-agnostic contrastive learning strategy for 3D segmentation. Since contrastive learning requires rich and diverse examples as keys and anchors, we leverage a prototype memory bank capturing class-wise global dataset information efficiently into a small number of prototypes acting as keys. An entropy-driven sampling technique then allows us to select good pixels from predictions as anchors. Experiments on three projection-based backbones show we outperform baselines on three challenging real-world outdoor datasets, working with as low as 0.001% annotations.

show abstract

“…A large part of recent work focuses on the case of reconstructing a single 3D scene given dense observations [16][17][18][19][20], enabling high-quality novel view synthesis with exciting applications in computer graphics. Alternatively, differentiable neural rendering may be used to supervise encoders to enable 3D reconstruction from incomplete image observations [21][22][23][24][25][26][27][28][29][30]. Fu and Zhang et al [31] use neural rendering as a tool to recover high-quality 2D panoptic segmentation annotations from a set of sparse images and noisy 3D bounding primitives and 2D predictions.…”

Section: Related Workmentioning

confidence: 99%

Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement

Sharma¹,

Tewari²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human perception reliably identifies movable and immovable parts of 3D scenes, and completes the 3D structure of objects and background from incomplete observations. We learn this skill not via labeled examples, but simply by observing objects move. In this work, we propose an approach that observes unlabeled multiview videos at training time and learns to map a single image observation of a complex scene, such as a street with cars, to a 3D neural scene representation that is disentangled into movable and immovable parts while plausibly completing its 3D structure. We separately parameterize movable and immovable scene parts via 2D neural ground plans. These ground plans are 2D grids of features aligned with the ground plane that can be locally decoded into 3D neural radiance fields. Our model is trained self-supervised via neural rendering. We demonstrate that the structure inherent to our disentangled 3D representation enables a variety of downstream tasks in street-scale 3D scenes using simple heuristics, such as extraction of objectcentric 3D representations, novel view synthesis, instance segmentation, and 3D bounding box prediction, highlighting its value as a backbone for data-efficient 3D scene understanding models. This disentanglement further enables scene editing via object manipulation such as deletion, insertion, and rigid-body motion.

show abstract

CoCoNets: Continuous Contrastive 3D Scene Representations

Cited by 10 publications

References 28 publications

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations

COARSE3D: Class-Prototypes for Contrastive Learning in Weakly-Supervised 3D Point Cloud Segmentation

Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement

Contact Info

Product

Resources

About