Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition

Hu, Lianyu; Liu, Shenglan; Feng, Wei

doi:10.48550/arxiv.2208.08599

Cited by 4 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recognizing that these simplistic blocks cannot adequately capture the data representation of a skeletal sequence, AutoGCN embraces the integration of prior knowledge. This algorithm aims to exploit the full potential of skeletal sequence information by defining the search space using architectural components known to be suitable for skeletal data [7], [9], [16], [21], [24], [34]. This departure from traditional methodologies acknowledges the need for a more nuanced and informed exploration of possible architectures in this domain [12], [15].…”

Section: Search Spacementioning

confidence: 99%

AutoGCN-Toward Generic Human Activity Recognition With Neural Architecture Search

Tempel,

Ihlen,

Strümke

2024

IEEE Access

View full text Add to dashboard Cite

This paper introduces AutoGCN, a generic Neural Architecture Search (NAS) algorithm for Human Activity Recognition (HAR) using Graph Convolution Networks (GCNs). HAR has enjoyed increased attention due to advances in deep learning, increased data availability, and enhanced computational capabilities. Concurrently, GCNs have shown promising abilities in modeling relationships between body key points in a skeletal graph. Typically, domain experts develop dataset-specific GCN-based methods, which limits their applicability beyond the specific context. AutoGCN seeks to address this limitation by simultaneously searching for the ideal hyperparameters and architecture combination within a versatile search space using a reinforcement controller while balancing optimal exploration and exploitation behavior with a knowledge reservoir during the search process. We conduct extensive experiments on two large datasets focused on skeleton-based action recognition to assess the proposed algorithm's performance. Our experimental results demonstrate the effectiveness of AutoGCN in constructing optimal GCN architectures for HAR, outperforming conventional NAS and GCN methods, as well as random search. These findings highlight the significance of a diverse search space and an expressive input representation to achieve good model performance and generalizability.INDEX TERMS Neural architecture search, human activity recognition, graph convolution networks, AutoML.

show abstract

Section: Search Spacementioning

confidence: 99%

AutoGCN-Toward Generic Human Activity Recognition With Neural Architecture Search

Tempel,

Ihlen,

Strümke

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The STGAT is proposed by Hu et al [77] to capture short-term dependencies of spatialtemporal modeling. STGAT uses the three simple modules to reduce local spatial-temporal feature redundancy and further release the potential.…”

Section: Gcn-basedmentioning

confidence: 99%

Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study

Nguyen¹,

Nguyen²,

Scherer

et al. 2023

Sensors

View full text Add to dashboard Cite

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human–machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal–spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid–DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid–DNN-based deep learning networks.

show abstract

“…Therefore, it is very popular to use pose estimation algorithms on RGB videos. Because of the simple but precise expression of motion information of skeletons and its stability to kinds of clothing textures and noisy backgrounds, there have been many methods trying to use skeleton data to solve action recognition problems [4,20,21]. Among them, the graph convolutional network (GCN) method is the most popular to deal with the graph structure of the skeletons, which treats joints as nodes and limbs as edges.…”

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

“…I3D [39] RGB 63.2 71.1 R(2+1)D-TwoStream [40] RGB, Flow -75.4 TRN [41] RGB 68.7 -TSM [42] RGB 70.6 -TSM Two-stream [42] RGB, Flow 81.2 -RSANet-R50 [38] RGB 86.4 -MARS [43] RGB,Flow -74.9 TP-ViT [11] RGB, Skeleton -80.8 STGAT [20] Skeleton…”

Section: Modality Finegym99 Kinetics400mentioning

confidence: 99%

A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition

et al. 2023

View full text Add to dashboard Cite

Due to the great success of Vision Transformer (ViT) in image classification tasks, many pure Transformer architectures for human action recognition have been proposed. However, very few works have attempted to use Transformer to conduct bimodal action recognition, i.e., both skeleton and RGB modalities for action recognition. As proved in many previous works, RGB modality and skeleton modality are complementary to each other in human action recognition tasks. How to use both RGB and skeleton modalities for action recognition in a Transformer-based framework is a challenge. In this paper, we propose RGBSformer, a novel two-stream pure Transformer-based framework for human action recognition using both RGB and skeleton modalities. Using only RGB videos, we can acquire skeleton data and generate corresponding skeleton heatmaps. Then, we input skeleton heatmaps and RGB frames to Transformer at different temporal and spatial resolutions. Because the skeleton heatmaps are primary features compared to the original RGB frames, we use fewer attention layers in the skeleton stream. At the same time, two ways are proposed to fuse the information of two streams. Experiments demonstrate that the proposed framework achieves the state of the art on four benchmarks: three widely used datasets, Kinetics400, NTU RGB+D 60, and NTU RGB+D 120, and the fine-grained dataset FineGym99.

show abstract

Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition

Cited by 4 publications

References 0 publications

AutoGCN-Toward Generic Human Activity Recognition With Neural Architecture Search

AutoGCN-Toward Generic Human Activity Recognition With Neural Architecture Search

Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study

A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition

Contact Info

Product

Resources

About