Human Skeleton Feature Optimizer and Adaptive Structure Enhancement Graph Convolution Network for Action Recognition

Xiong, Xin; Min, Weidong; Wang, Qi; Zha, Cheng

doi:10.1109/tcsvt.2022.3201186

Cited by 14 publications

(4 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cross Subject(%) Cross View(%) X-sub120 X-set120 Param(M) ST-GCN (Yan, Xiong, and Lin 2018) 81.5 88.3 --3.10 AS-GCN (Li et (Li et al 2019) 34.8 56.5 2S-AGCN (Shi et al 2019b) 36.1 58.7 MS-G3D 38.0 60.9 MST-GCN (Chen et al 2021d) 38.1 60.8 SMotif-GCN+TBs (Wen et al 2022) 37.8 60.6 ASE-GCN (Xiong et al 2022) 36.9 59.7 ML-STGNet (Zhu et al 2022) 38.9 62.2 2M-STGCN (Zhang et al 2023) 39.0 61.6 STF-Net (Wu, Zhang, and Zou 2023) 36.1 58.9 FRF-GCN(ours) 37.9 60.7…”

Section: Algorithmmentioning

confidence: 99%

“…Currently, most behavior recognition models based on skeleton data employ architectures that fuse multiple streams of information (Hu et al 2022;Qin et al 2022;Wu, Zhang, and Zou 2023;Zhang et al 2023). The data fusion schemes used can be broadly classified into two categories: 1) Rear fusion of data (Chen et al 2021d;Liu et al 2022a;Xiong et al 2022), where each stream of information is processed by the same model to obtain behavior category scores. These scores are then combined through weighted fusion to produce a final behavior classification result.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy

Yun,

Xu,

Riou

et al. 2024

AAAI

View full text Add to dashboard Cite

The deployment of multi-stream fusion strategy on behavioral recognition from skeletal data can extract complementary features from different information streams and improve the recognition accuracy, but suffers from high model complexity and a large number of parameters. Besides, existing multi-stream methods using a fixed adjacency matrix homogenizes the model’s discrimination process across diverse actions, causing reduction of the actual lift for the multi-stream model. Finally, attention mechanisms are commonly applied to the multi-dimensional features, including spatial, temporal and channel dimensions. But their attention scores are typically fused in a concatenated manner, leading to the ignorance of the interrelation between joints in complex actions. To alleviate these issues, the Front-Rear dual Fusion Graph Convolutional Network (FRF-GCN) is proposed to provide a lightweight model based on skeletal data. Targeted adjacency matrices are also designed for different front fusion streams, allowing the model to focus on actions of varying magnitudes. Simultaneously, the mechanism of Spatial-Temporal-Channel Parallel Attention (STC-P), which processes attention in parallel and places greater emphasis on useful information, is proposed to further improve model’s performance. FRF-GCN demonstrates significant competitiveness compared to the current state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120 and Kinetics-Skeleton 400 datasets. Our code is available at: https://github.com/sunbeam-kkt/FRF-GCN-master.

show abstract

Section: Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy

Yun,

Xu,

Riou

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…(3) It is a directed and acyclic graph and gathers all information to the center nose joint. As mentioned in [36] and [37], the GCNbased model becomes over smoothing as the increase in the number of network layers since the node representation in the same connected component tends to converge to the same value for information aggregation and update. The directed and acyclic graph makes the information flow in one direction and relieves the over-smoothing issue in GCN learning.…”

Section: B Skeleton Graph Designmentioning

confidence: 99%

SA-BiGCN: Bi-Stream Graph Convolution Networks With Spatial Attentions for the Eye Contact Detection in the Wild

Ling,

Ma,

Xie

et al. 2024

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

Eye contact is essential in transmitting information and intention in the wild environment (e.g., urban streets or parking lots) with mixed vehicles and pedestrians. Compared with the vision image data, the human skeleton data are deemed to be robust to unconstrained surroundings and illumination. However, the skeleton graph-based approaches are mainly used for the action recognition. It is challenging to directly apply them to the eye detection task, which is momentary and dynamic given the complex wild environment. This paper proposes a Bi-stream Spatial Attention Graph Convolution Network (SA-BiGCN) for eye contact detection in the wild. We design a directed, nose-centric skeleton graph to capture relevant and hierarchical information and their interactions. We also propose a Bi-stream graph convolution network model with spatial attention to dynamically extract and fuse skeleton joints and bones information. The model was validated by comparing with state-of-art models on three large-scale public datasets, including JAAD, PIE, and LOOK. The results highlight the accuracy and generalization performance of the proposed SA-BiGCN model in detecting the eye contact in the wild environment. The ablation analysis validates the importance of the skeleton graph design, the spatial attention mechanism in the feature fusion process, as well as the model robustness against noisy skeleton data in terms of part occlusions, block occlusions, random occlusions, and random deviations.

show abstract

“…The core of CNN is the computation between filter kernels and features, where new feature information can be extracted from input features by convolution operations. Vanilla convolution is widely used in deep learning-based image inpainting methods to extract features by successively sliding a window over all pixels of an image [17][18][19]. Context Encoders [20] is one of the early network models to complete the image inpainting task by training a CNN consisting of vanilla convolutions.…”

Section: Related Workmentioning

confidence: 99%

Image inpainting network based on multi‐level attention mechanism

Xiang,

Min,

Wei

et al. 2023

IET Image Processing

Self Cite

View full text Add to dashboard Cite

Image inpainting networks based on deep learning techniques have been widely used in many important fields. However, most inpainting networks fail to generate desirable repaired images. This may be due to their failure to extract effective features and accurately assign high weights to the undamaged regions. To alleviate these problems, an image inpainting network based on gated convolution and multi‐level attention mechanism (IIN‐GCMAM) is proposed in this paper. This network follows encoder–decoder architecture, consisting of the gated convolution encoder (GC‐encoder) and the multi‐level attention mechanism decoder (MAM‐decoder). The GC‐encoder weighs the extracted features with gated convolutions, which reduces the interference caused by the damaged regions. The multi‐level attention mechanism employed in the MAM‐decoder uses multi‐scale feature maps spatially and channel‐wise to improve the consistency in global structure and the fineness of repaired results. Extensive experiments are conducted on the common datasets, Paris StreetView and CelebA. Experimental results indicate that the proposed IIN‐GCMAM can achieve a good performance on the common evaluation metrics and visual effects. It can achieve 0.0408, 0.720, and 22.27 in MAE, SSIM, and PSNR at the mask ratio of 50%–60%, respectively.

show abstract

Human Skeleton Feature Optimizer and Adaptive Structure Enhancement Graph Convolution Network for Action Recognition

Cited by 14 publications

References 61 publications

Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy

Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy

SA-BiGCN: Bi-Stream Graph Convolution Networks With Spatial Attentions for the Eye Contact Detection in the Wild

Image inpainting network based on multi‐level attention mechanism

Contact Info

Product

Resources

About