Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding

Hao, Wen; Liu, Yunze; Huang, Jingwei; Duan, Baoxing; Yi, Li

doi:10.1007/978-3-031-19818-2_2

Cited by 13 publications

(9 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dataset provides annotations for panoptic and motion segmentation, 3D hand pose, category-level object pose and includes reconstructed object meshes and scene point clouds. This dataset has proven useful for object segmentation and shape reconstruction (Liu et al, 2023c;Zhang et al, 2023d;Wen et al, 2022), action segmentation (Reza et al, 2023;Zhang et al, 2023d), hand-object manipulation synthesis (Zheng et al, 2023a;Ye et al, 2023b), hand action detection (Hung-Cuong et al, 2023) and 3D hand pose estimation (Ye et al, 2023b).…”

Section: General Datasetsmentioning

confidence: 99%

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

What will the future be? We wonder! In this survey, we explore the gap between current research in egocentric vision and the ever-anticipated future, where wearable computing, with outward facing cameras and digital overlays, is expected to be integrated in our every day lives. To understand this gap, the article starts by envisaging the future through character-based stories, showcasing through examples the limitations of current technology. We then provide a mapping between this future and previously defined research tasks. For each task, we survey its seminal works, current state-of-the-art methodologies and available datasets, then reflect on shortcomings that limit its applicability to future research. Note that this survey focuses on software models for egocentric vision, independent of any specific hardware. The paper concludes with recommendations for areas of immediate explorations so as to unlock our path to the future always-on, personalised and life-enhancing egocentric vision.

show abstract

Section: General Datasetsmentioning

confidence: 99%

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

show abstract

“…Each frame is sampled by 2,048 points. We maintain the same training/testing split as previous works (Wen et al 2022;Zhang et al 2023). Evaluation metrics.…”

Section: Experiments Experiments Setupmentioning

confidence: 99%

“…Comparison with State-of-the-arts HOI4D action segmentation. Table 1 demonstrates the results on HOI4D dataset for the task of action segmentation, where we compare our method with previously published methods (Fan and Kankanhalli 2021;Wen et al 2022;Zhang et al 2023) rics, on both test and validation sets. The test set results are sourced from the HOI4D online leaderboard.…”

Section: Experiments Experiments Setupmentioning

confidence: 99%

“…Exploring point cloud sequences in 4D (integrating 3D space with 1D time) has garnered considerable interest in recent years (Fan and Kankanhalli 2021;Wen et al 2022;Liu et al 2022) due to their capacity to offer a wealth of dynamic information within our 3D environment. Compared to conventional videos, 4D point clouds deliver direct access to geometric information in 3D space, a facet particularly advantageous for real-world interactions.…”

Section: Introductionmentioning

confidence: 99%

“…Previous works in 4D point cloud representation learning predominantly stem from extending existing 3D point cloud models (Fan and Kankanhalli 2021;Wen et al 2022) to 4D, which involves incorporating additional temporal learning modules that enable feature interactions across time (Xiao et al 2022;Zhong et al 2022). However, due to the sparsity and lack of texture in point clouds, these methods are limited in capturing comprehensive semantic details.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer

Jing,

Xue,

Yan

et al. 2024

AAAI

View full text Add to dashboard Cite

The field of 4D point cloud understanding is rapidly developing with the goal of analyzing dynamic 3D point cloud sequences. However, it remains a challenging task due to the sparsity and lack of texture in point clouds. Moreover, the irregularity of point cloud poses a difficulty in aligning temporal information within video sequences. To address these issues, we propose a novel cross-modal knowledge transfer framework, called X4D-SceneFormer. This framework enhances 4D-Scene understanding by transferring texture priors from RGB sequences using a Transformer architecture with temporal relationship mining. Specifically, the framework is designed with a dual-branch architecture, consisting of an 4D point cloud transformer and a Gradient-aware Image Transformer (GIT). The GIT combines visual texture and temporal correlation features to offer rich semantics and dynamics for better point cloud representation. During training, we employ multiple knowledge transfer techniques, including temporal consistency losses and masked self-attention, to strengthen the knowledge transfer between modalities. This leads to enhanced performance during inference using single-modal 4D point cloud inputs. Extensive experiments demonstrate the superior performance of our framework on various 4D point cloud video understanding tasks, including action recognition, action segmentation and semantic segmentation. The results achieve 1st places, i.e., 85.3% (+7.9%) accuracy and 47.3% (+5.0%) mIoU for 4D action segmentation and semantic segmentation, on the HOI4D challenge, outperforming previous state-of-the-art by a large margin. We release the code at https://github.com/jinglinglingling/X4D.

show abstract

Applications of 4D Point Clouds (4DPC) in Digital Twin Construction: A SWOT Analysis

Dong

Xue

2023

Lecture Notes in Operations Research

View full text Add to dashboard Cite

Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding

Cited by 13 publications

References 29 publications

An Outlook into the Future of Egocentric Vision

An Outlook into the Future of Egocentric Vision

X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer

Applications of 4D Point Clouds (4DPC) in Digital Twin Construction: A SWOT Analysis

Contact Info

Product

Resources

About