Transformers in 3D Point Clouds: A Survey

Abstract

In recent years, Transformer models have been proven to have the remarkable ability of long-range dependencies modeling. They have achieved satisfactory results both in Natural Language Processing (NLP) and image processing. This significant achievement sparks great interest among researchers in 3D point cloud processing to apply them to various 3D tasks. Due to the inherent permutation invariance and strong global feature learning ability, 3D Transformers are well suited for point cloud processing and analysis. They have achieved competitive or even better performance compared to the state-of-the-art non-Transformer algorithms. This survey aims to provide a comprehensive overview of 3D Transformers designed for various tasks (e.g. point cloud classification, segmentation, object detection, and so on). We start by introducing the fundamental components of the general Transformer and providing a brief description of its application in 2D and 3D fields. Then, we present three different taxonomies (i.e., Transformer implementation-based taxonomy, data representation-based taxonomy, and task-based taxonomy) for method classification, which allows us to analyze involved methods from multiple perspectives. Furthermore, we also conduct an investigation of 3D self-attention mechanism variants designed for performance improvement. To demonstrate the superiority of 3D Transformers, we compare the performance of Transformer-based algorithms in terms of point cloud classification, segmentation, and object detection. Finally, we point out three potential future research directions, expecting to provide some benefit references for the development of 3D Transformers.

Combining Image and Point Cloud Segmentation to Improve Heritage Understanding

“…Early fusion methods, for instance, involve rendering 3D information as multi-view 2D images with an additional depth channel (RGBD), which can then be processed by standard 2D convolutions (Cui et al, 2022) (MVCNN). Alternatively, 2D images can be rendered as a 3D graph, tree, or raster point cloud representation (Lu et al, 2022). However, these 2D methods often lose some 3D geometric context and struggle with per-point label prediction.…”

“…Recent advancements in MVCNN networks include ShapeConv (Cao et al, 2021) and FPS-Net (Xiao et al, 2021). On the other hand, late fusion combines the outputs of multiple networks and averages the results, for example, by integrating Point Transformers (Lu et al, 2022) with purely image-based networks. The advantage here is that each modality can be trained separately, leveraging numerous available benchmarks.…”

6-DoF Robotic Grasping with Transformer

“…PointNet mainly relies on local feature learning to aggregate global information progressively, it is still not efficient for robotic grasping, which requires an effective encoding of the global information of an input. In computer vision and graphics, researchers have explored the use of transformer models on point cloud processing [29], such as point cloud segmentation [20], classification [30], and shape completion [31]. However, the number of points in a point cloud input is not fixed and too high to be processed with multi-head attention efficiently, which is especially serious in a real robot scenario.…”

A survey on deep reinforcement learning for audio-based applications

“…Additionally, positional encoding conveys information about token positions (see Figure 2). These benefits have spurred significant interest in transformers across various AI domains [76][77][78][79][80][81][82], notably the audio community. This has given rise to diverse architectures such as Wav2Vec [83], Whisper [84], FastPitch [85], MusicBERT [86], and others [26,87,88].…”