STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Zhou, Kanglei; Cheng, Zhiyuan; Shum, Hubert P. H.; Li, Frederick W. B.; Liang, Xiaohui

doi:10.1109/ismar52148.2021.00018

Cited by 4 publications

(14 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(3) We have conducted new experiments to fully evaluate the proposed system with an additional dataset, SHREC, alongside the NYU dataset, to provide a comprehensive and diverse evaluation of our system. We evidence the mutual benefit of the two tasks and the performance advancement over the state-of-theart, including [59]. (4) We have conducted new qualitative experiments, which are presented in the paper as well as a newly created video.…”

Section: Introductionmentioning

confidence: 93%

“…Motivated by the success of STGCN, we propose a novel approach called STGAE [59] that combines STGCN with a hand skeleton partitioning strategy based on hand symmetry priors to effectively denoise hand motion data. However, there is a risk of over-smoothing the results and losing temporal dynamics with existing STGCN-based methods.…”

Section: Graph Convolutional Networkmentioning

confidence: 99%

“…W ITH the rapid advancement of Human-Computer Interaction (HCI) techniques, human hands play a vital role in performing operations such as grasping and manipulating objects [42], [58]. In many HCI applications [57] such as Virtual/Augmented Reality (VR/AR), providing an immersive experience for users relies on quickly tracking human hands and accurately estimating the corresponding hand poses [27], [40].…”

Section: Introductionmentioning

confidence: 99%

“…Our preliminary results of a denoising-only method have been presented in [59]. This paper has made significant advancements and presents the following technical novelty.…”

Section: Introductionmentioning

confidence: 99%

“…(2) We have included new materials in the paper, including the design and justification to explain the new multi-task framework, with a light touch to explain how individual components of [59] are adapted. (3) We have conducted new experiments to fully evaluate the proposed system with an additional dataset, SHREC, alongside the NYU dataset, to provide a comprehensive and diverse evaluation of our system.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Zhou,

Shum,

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

In many human-computer interaction applications, fast and accurate hand tracking is necessary for an immersive experience. However, raw hand motion data can be flawed due to issues such as joint occlusions and high-frequency noise, hindering the interaction. Using only current motion for interaction can lead to lag, so predicting future movement is crucial for a faster response. Our solution is the Multi-task Spatial-Temporal Graph Auto-Encoder (Multi-STGAE), a model that accurately denoises and predicts hand motion by exploiting the inter-dependency of both tasks. The model ensures a stable and accurate prediction through denoising while maintaining motion dynamics to avoid over-smoothed motion and alleviate time delays through prediction. A gate mechanism is integrated to prevent negative transfer between tasks and further boost multi-task performance. Multi-STGAE also includes a spatial-temporal graph autoencoder block, which models hand structures and motion coherence through graph convolutional networks, reducing noise while preserving hand physiology. Additionally, we design a novel hand partition strategy and hand bone loss to improve natural hand motion generation. We validate the effectiveness of our proposed method by contributing two large-scale datasets with a data corruption algorithm based on two benchmark datasets. To evaluate the natural characteristics of the denoised and predicted hand motion, we propose two structural metrics. Experimental results show that our method outperforms the state-of-the-art, showcasing how the multitask framework enables mutual benefits between denoising and prediction.

show abstract

Section: Introductionmentioning

confidence: 93%

Section: Graph Convolutional Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Our preliminary results of a denoising-only method have been presented in [59]. This paper has made significant advancements and presents the following technical novelty.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Zhou,

Shum,

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

A Mixed Reality Training System for Hand-Object Interaction in Simulated Microgravity Environments

Zhou,

Chen,

et al. 2023

2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Self Cite

View full text Add to dashboard Cite

show abstract

Hierarchical Graph Convolutional Networks for Action Quality Assessment

Zhou

Shum

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

Self Cite

View full text Add to dashboard Cite

Action quality assessment (AQA) automatically evaluates how well humans perform actions in a given video, a technique widely used in fields such as rehabilitation medicine, athletic competitions, and specific skills assessment. However, existing works that uniformly divide the video sequence into small clips of equal length suffer from intra-clip confusion and inter-clip incoherence, hindering the further development of AQA. To address this issue, we propose a hierarchical graph convolutional network (GCN). First, semantic information confusion is corrected through clip refinement, generating the 'shot' as the basic action unit. We then construct a scene graph by combining several consecutive shots into meaningful scenes to capture local dynamics. These scenes can be viewed as different procedures of a given action, providing valuable assessment cues. The video-level representation is finally extracted via sequential action aggregation among scenes to regress the predicted score distribution, enhancing discriminative features and improving assessment performance. Experiments on the AQA-7, MTL-AQA, and JIGSAWS datasets demonstrate the superiority of the proposed hierarchical GCN over state-of-the-art methods.

show abstract

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Cited by 4 publications

References 32 publications

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

A Mixed Reality Training System for Hand-Object Interaction in Simulated Microgravity Environments

Hierarchical Graph Convolutional Networks for Action Quality Assessment

Contact Info

Product

Resources

About