DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

Wang, Haiyang; Chen, Shi; Shi, Shaoshuai; Li, Meng; Wang, Sen; He, Dawei; Schiele, Bernt; Wang, Liwei

doi:10.48550/arxiv.2301.06051

Cited by 2 publications

(2 citation statements)

References 37 publications

(94 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the recent progress in transformers for computer vision [10], the transformer architecture has also been applied to 3D object detection. Existing works include transformer-based backbones [44,22,12,65,13,73] for the voxel-based representation, [50,91] for the point-based representation, and [21] for a combination of both the point and voxel representation. Furthermore, transformers have been used to improve the detection head [97] and for sensor fusion [2,90,76].…”

Section: Lidar-based 3d Object Detectionmentioning

confidence: 99%

Efficient Transformer-based 3D Object Detection with Dynamic Token Halting

Ye¹,

Meyer²,

Chai³

et al. 2023

Preprint

View full text Add to dashboard Cite

Balancing efficiency and accuracy is a long-standing problem for deploying deep learning models. The tradeoff is even more important for real-time safety-critical systems like autonomous vehicles. In this paper, we propose an effective approach for accelerating transformer-based 3D object detectors by dynamically halting tokens at different layers depending on their contribution to the detection task. Although halting a token is a non-differentiable operation, our method allows for differentiable end-to-end learning by leveraging an equivalent differentiable forward-pass. Furthermore, our framework allows halted tokens to be reused to inform the model's predictions through a straightforward token recycling mechanism. Our method significantly improves the Pareto frontier of efficiency versus accuracy when compared with the existing approaches. By halting tokens and increasing model capacity, we are able to improve the baseline model's performance without increasing the model's latency on the Waymo Open Dataset.

show abstract

Section: Lidar-based 3d Object Detectionmentioning

confidence: 99%

Efficient Transformer-based 3D Object Detection with Dynamic Token Halting

Ye¹,

Meyer²,

Chai³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The cost of structuring data, nevertheless, turned out to be a performance bottleneck for large inputs. Recently, a dynamic sparse voxel transformer (DSVT) was presented by Wang et al ( 2023 ) in an effort to widen the uses of transformers so that they may serve as a solid foundation for outdoor 3D perception just as they do for 2D vision. A number of local regions are split up into smaller ones in each window using DSVT based on sparsity, and each window's attributes are then computed fully in parallel.…”

Section: Related Workmentioning

confidence: 99%

An improved fused feature residual network for 3D point cloud data

Gezawa,

Liu,

Jia

et al. 2023

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Point clouds have evolved into one of the most important data formats for 3D representation. It is becoming more popular as a result of the increasing affordability of acquisition equipment and growing usage in a variety of fields. Volumetric grid-based approaches are among the most successful models for processing point clouds because they fully preserve data granularity while additionally making use of point dependency. However, using lower order local estimate functions to close 3D objects, such as the piece-wise constant function, necessitated the use of a high-resolution grid in order to capture detailed features that demanded vast computational resources. This study proposes an improved fused feature network as well as a comprehensive framework for solving shape classification and segmentation tasks using a two-branch technique and feature learning. We begin by designing a feature encoding network with two distinct building blocks: layer skips within, batch normalization (BN), and rectified linear units (ReLU) in between. The purpose of using layer skips is to have fewer layers to propagate across, which will speed up the learning process and lower the effect of gradients vanishing. Furthermore, we develop a robust grid feature extraction module that consists of multiple convolution blocks accompanied by max-pooling to represent a hierarchical representation and extract features from an input grid. We overcome the grid size constraints by sampling a constant number of points in each grid using a simple K-points nearest neighbor (KNN) search, which aids in learning approximation functions in higher order. The proposed method outperforms or is comparable to state-of-the-art approaches in point cloud segmentation and classification tasks. In addition, a study of ablation is presented to show the effectiveness of the proposed method.

show abstract

DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

Cited by 2 publications

References 37 publications

Efficient Transformer-based 3D Object Detection with Dynamic Token Halting

Efficient Transformer-based 3D Object Detection with Dynamic Token Halting

An improved fused feature residual network for 3D point cloud data

Contact Info

Product

Resources

About