Graph Based Skeleton Modeling for Human Activity Analysis

Kao, Jiun-Yu; Ortega, Antonio; Tian, Dong; Mansour, Hassan; Vetro, Anthony

doi:10.1109/icip.2019.8803186

Cited by 22 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many HAR systems are based on datasets that combine both RGB color and depth information [44]. RGB-D sensors also provide skeletal information [45]. Table 3 shows the details of authors, datasets and research work based on RGB-D sensors, using the combination of both RGB and depth images.…”

Section: Rgb-d-based Har Systemsmentioning

confidence: 99%

Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

et al. 2021

View full text Add to dashboard Cite

Due to the constantly increasing demand for automatic tracking and recognition systems, there is a need for more proficient, intelligent and sustainable human activity tracking. The main purpose of this study is to develop an accurate and sustainable human action tracking system that is capable of error-free identification of human movements irrespective of the environment in which those actions are performed. Therefore, in this paper we propose a stereoscopic Human Action Recognition (HAR) system based on the fusion of RGB (red, green, blue) and depth sensors. These sensors give an extra depth of information which enables the three-dimensional (3D) tracking of each and every movement performed by humans. Human actions are tracked according to four features, namely, (1) geodesic distance; (2) 3D Cartesian-plane features; (3) joints Motion Capture (MOCAP) features and (4) way-points trajectory generation. In order to represent these features in an optimized form, Particle Swarm Optimization (PSO) is applied. After optimization, a neuro-fuzzy classifier is used for classification and recognition. Extensive experimentation is performed on three challenging datasets: A Nanyang Technological University (NTU) RGB+D dataset; a UoL (University of Lincoln) 3D social activity dataset and a Collective Activity Dataset (CAD). Evaluation experiments on the proposed system proved that a fusion of vision sensors along with our unique features is an efficient approach towards developing a robust HAR system, having achieved a mean accuracy of 93.5% with the NTU RGB+D dataset, 92.2% with the UoL dataset and 89.6% with the Collective Activity dataset. The developed system can play a significant role in many computer vision-based applications, such as intelligent homes, offices and hospitals, and surveillance systems.

show abstract

Section: Rgb-d-based Har Systemsmentioning

confidence: 99%

Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

et al. 2021

View full text Add to dashboard Cite

show abstract

“…We pick one action "hand waving", which includes 55 frames. Each graph signal is the 3D coordinates supported on a skeleton graph recorded in each frame [25].…”

Section: Datasetmentioning

confidence: 99%

Time-Varying Graph Signal Inpainting Via Unrolling Networks

Chen

Eldar

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We propose an interpretable graph neural network based on algorithm unrolling to reconstruct a time-varying graph signal from partial measurements. The proposed graph unrolling networks expand algorithm unrolling to the graph-time domain and provide an interpretation of the architecture design from a signal processing perspective. We unroll an iterative inpainting algorithm by mapping each iteration to a single network layer. The feed-forward process is thus equivalent to iteratively reconstructing a time-varying graph signal. We train this network through unsupervised learning, where the input time-varying graph signal is used to supervise the training. By leveraging the learning ability of neural networks, we adaptively capture appropriate priors from input data, instead of manually choosing signal priors. To validate the proposed methods, we conduct experiments on three real-world datasets and demonstrate that our networks achieve smaller reconstruction errors than conventional inpainting algorithms and state-of-the-art graph neural networks.

show abstract

“…The literature of action understanding from full-body skeleton data includes methods such as co-occurrence feature learning [8], spatialtemporal graph convolutional networks [9], spectral graph-based skeleton modeling [10], [11]. The success of deep learning methods has led to a surge of deep learning-based skeleton modeling methods in the past few years.…”

Section: Introductionmentioning

confidence: 99%

“…Our proposed approach, Symmetric Sub-graph-spatio-temporal graph convolutional networks (S 2 -ST-GCN), is shown to achieve better accuracy with fewer convolutional layers. Our approach is based on the observation that skeleton graphs [11], including hand graphs [19], are symmetric in nature. Symmetry is not only useful computationally, as shown in [21], but also closely tied to human motion, so that exploiting it can lead to better recognition.…”

Section: Introductionmentioning

confidence: 99%

Symmetric Sub-graph Spatio-Temporal Graph Convolution and its application in Complex Activity Recognition

Das

Ortega

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Understanding complex hand actions, such as assembly tasks or kitchen activities, from hand skeleton data is an important yet challenging task. In this paper, we analyze hand skeleton-based complex activities by modeling dynamic hand skeletons through a spatiotemporal graph convolutional neural network (ST-GCN). This model jointly learns and extracts Spatio-temporal features for activity recognition. Our proposed technique, Symmetric Sub-graph spatio-temporal graph convolutional neural network (S 2 -ST-GCN), exploits the symmetric nature of hand graphs to decompose them into smaller sub-graphs, which allow us to build a separate temporal model for the relative motion of the fingers. This subgraph approach can be implemented efficiently by preprocessing input data using a Haar unit based orthogonal matrix. Then, in addition to spatial filters, separate temporal filters can be learned for each sub-graph. We evaluate the performance of the proposed method on the First-Person Hand Action dataset. While the proposed method shows comparable performance with the state of the art methods in train:test=1:1 setting, it achieves this with greater stability. Furthermore, we demonstrate significant performance improvement in comparison to state of the art methods in the cross-person setting, where the model did not come across a test subject's data while learning. S 2 -ST-GCN also shows superior performance than a finger-based decomposition of the hand graph where no preprocessing is applied.Index Terms-Hand Skeleton, Graph based methods, complex activity analysis, Spatio-temporal graph neural network, First person hand action (FPHA) dataset

show abstract

Graph Based Skeleton Modeling for Human Activity Analysis

Cited by 22 publications

References 15 publications

Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

Time-Varying Graph Signal Inpainting Via Unrolling Networks

Symmetric Sub-graph Spatio-Temporal Graph Convolution and its application in Complex Activity Recognition

Contact Info

Product

Resources

About