Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition

Guo, Zizhao; Ying, Sancong

doi:10.3390/app12126215

Cited by 2 publications

(3 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Video recognition [29][30][31] is an important direction in computer vision and video processing research. To date, many effective video recognition methods have been developed, which can be grouped into two categories: temporal-spatial-and spatial-based video recognition methods.…”

Section: Discussionmentioning

confidence: 99%

Manifolds-Based Low-Rank Dictionary Pair Learning for Efficient Set-Based Video Recognition

et al. 2023

View full text Add to dashboard Cite

As an important research direction in image and video processing, set-based video recognition requires speed and accuracy. However, the existing static modeling methods focus on computational speed but ignore accuracy, whereas the dynamic modeling methods are higher-accuracy but ignore the computational speed. Combining these two types of methods to obtain fast and accurate recognition results remains a challenging problem. Motivated by this, in this study, a novel Manifolds-based Low-Rank Dictionary Pair Learning (MbLRDPL) method was developed for a set-based video recognition/image set classification task. Specifically, each video or image set was first modeled as a covariance matrix or linear subspace, which can be seen as a point on a Riemannian manifold. Second, the proposed MbLRDPL learned discriminative class-specific synthesis and analysis dictionaries by clearly imposing the nuclear norm on the synthesis dictionaries. The experimental results show that our method achieved the best classification accuracy (100%, 72.16%, 95%) on three datasets with the fastest computing time, reducing the errors of state-of-the-art methods (JMLC, DML, CEBSR) by 0.96–75.69%.

show abstract

Section: Discussionmentioning

confidence: 99%

Manifolds-Based Low-Rank Dictionary Pair Learning for Efficient Set-Based Video Recognition

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Guo, Z. et al (2022) [3] suggested a method for capturing detailed body information using Whole-Body Keypoint and Skeleton (WKS) labels. They developed an architecture that achieves superior performance by utilizing the Swin transformer when combined with three-dimensional (3D) convolutional neural networks (CNNs) to extract spatiotemporal characteristics.…”

Section: Related Workmentioning

confidence: 99%

“…The initial model structure was specifically designed for video data, incorporating multiple 3D convolutional layers to capture both spatial and temporal features effectively. The input shape here is (25,224,224,3) which represents (the number of frames per sequence, image height, image width, and color channel). At a certain point, it could predict all the training datasets.…”

Section: Proposed Model Evaluation 61 Experimental 3dcnn Modelmentioning

confidence: 99%

Dynamic BdSL Digit Recognition using Hybrid 3DCNN+BiLSTM Model

2024

IJFMR

View full text Add to dashboard Cite

Sign language is used as a way of communication by deaf and mute individuals. However, due to the limited number of people who understand sign language, integrating them into society is challenging. Approximately 6.9% of Bangladesh's population and 5% of the world’s population suffer from speech impediments. Individuals with this condition cannot hear what others are saying or communicate verbally, thus sign language must be relied upon. In recent years, sign language recognition has gained attention due to its necessity. However, a scarcity of publicly available dynamic gesture datasets for Bangladeshi Sign Language (BdSL) exists. Dynamic gestures, which contain both spatial and temporal information, are more useful in real-life applications. The classification of dynamic gestures requires more data than static gestures. In this research, 11 numeral Bengali digit signs are aimed to be classified. Data has been collected from the “SignBD-Word” dataset, which contains extracted RGB human body pose keypoints skeleton data. When compared to other dynamic sign language datasets and data requirements for training deep neural networks for dynamic gestures, the data per class in this dataset is found to be insufficient. A hybrid model architecture is proposed in this research to recognize dynamic gestures using lightweight 3DCNN and bidirectional LSTM layers for classifying gestures from the motion patterns of human body pose keypoints skeleton data after experimenting on various models. It has been observed that combining 3DCNN with the pre-trained DenseNet-201 and the BiLSTM model increases real-time accuracy by 4.54%. To the best of our knowledge, this is the first approach to combine 3DCNN with DenseNet-201 for action recognition. Also, one of the earliest investigations on dynamic BdSL hand gesture digit recognition using dynamic data. Additionally, different pre-trained models as base feature extractors have been evaluated.

show abstract

Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition

Cited by 2 publications

References 50 publications

Manifolds-Based Low-Rank Dictionary Pair Learning for Efficient Set-Based Video Recognition

Manifolds-Based Low-Rank Dictionary Pair Learning for Efficient Set-Based Video Recognition

Dynamic BdSL Digit Recognition using Hybrid 3DCNN+BiLSTM Model

Contact Info

Product

Resources

About