HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm

Islam, Md Mofijul; Iqbal, Tarıq

doi:10.1109/iros45743.2020.9340987

Cited by 54 publications

(28 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From Table 1, we can see that the proposed SRCA achieves the best result 96.37%. Compared with HMLAT [25] which is one of the most effective methods currently, SRCA outperforms by 1.31%. Compared with methods [4,22,23,24] based on handcraft features and deep learning, SRCA based on deep learning also is superior without complex preprocessing to obtain handcraft features.…”

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 99%

Spatial Reasoning and Context-Aware Attention Network for Skeleton-Based Action Recognition

You

Han

Liang

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Skeleton-based action recognition has achieved promising performance recently, but there are still many challenges, e.g., the structural relation between joints and the different attention of frames, due to the complex spatial-temporal evolution of skeletal joints. In this paper, we propose a spatial reasoning and context-aware attention network for skeleton-based action recognition, which consists of a spatial reasoning module and a context-aware attention module. The spatial reasoning module can exploit the structural relation between joints to obtain the spatial features within each skeleton frame, followed by the context-aware attention module learning the different attention of frames. We perform experiments on two datasets and verify the effectiveness of each module in the proposed network. The comparison results demonstrate that our method achieves state-of-the-art performance.

show abstract

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 99%

Spatial Reasoning and Context-Aware Attention Network for Skeleton-Based Action Recognition

You

Han

Liang

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

show abstract

“…This simple method can be applied to RNN and LSTM. Islam et al [28] proposed a hierarchical multi-modal attention-based human activity recognition algorithm. It uses N modalities for action recognition.…”

Section: B Human Action Recognitionmentioning

confidence: 99%

Learning 3D Skeletal Representation From Transformer for Action Recognition

et al. 2022

View full text Add to dashboard Cite

Skeleton-based human action recognition has attracted significant interest due to its simplicity and good accuracy. Diverse end-to-end trainable frameworks based on skeletal representation have been proposed so far to map the representation to human action classes better. Most skeleton-based human action recognition approaches are based on the skeletons, which are heuristically pre-defined by the commercial sensors. Nevertheless, it is not confirmed whether the sensor-captured skeletons is the best representation of human bodies for the action recognition task, while in general, the dedicated representation is required for achieving the successful performance on subsequent tasks such as action recognition. In this paper, we try to deal with the issue by explicitly learning the skeletal representation in the context of the human action recognition task. We start our investigation by reconstructing 3D meshes of the human bodies from RGB videos. Then we involve the transformer architecture to sample the most informative skeletal representation from reconstructed 3D meshes, considering the inner and inter structural relationship of 3D meshes and sensor-captured skeletons. Experimental results on challenging human action recognition benchmarks (i.e., SYSU and UTD-MHAD datasets) have shown the superiority of our skeletal representation compared to the sensor-captured skeletons for the action recognition task.INDEX TERMS 3D representation, action recognition, human mesh, transformer.

show abstract

“…[29] and a variety of approaches developed specifically for robotics [4], [5], [6], [8], [10], deploying such models in practice is very hard, since robots often operate in dynamic environments where changes of potential activities may occur at any time. Still, the majority of previously published research strives for high accuracy on conventionl ADL recognition datasets [5], [26], [30] assuming that a large amount of labelled training examples is available for every activity-of-interest.…”

Section: Related Workmentioning

confidence: 99%

“…There have been impressive advances activity recognition frameworks tailored for robotics applications [1], [2], [3], [4], [5], [6], [7], [8]. However, this task remains very challenging in practice, as agents mostly operate in an open constantly changing environment and we will never be able to capture and annotate a high amount of training examples for every possible category [9], which is a requirement in the majority of presented approaches.…”

Section: Introductionmentioning

confidence: 99%

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatically understanding human behaviour allows household robots to identify the most critical needs and plan how to assist the human according to the current situation. However, the majority of such methods are developed under the assumption that a large amount of labelled training examples is available for all concepts-of-interest. Robots, on the other hand, operate in constantly changing unstructured environments, and need to adapt to novel action categories from very few samples. Methods for data-efficient recognition from body poses increasingly leverage skeleton sequences structured as image-like arrays and then used as input to convolutional neural networks. We look at this paradigm from the perspective of transformer networks, for the first time exploring visual transformers as data-efficient encoders of skeleton movement. In our pipeline, body pose sequences cast as image-like representations are converted into patch embeddings and then passed to a visual transformer backbone optimized with deep metric learning. Inspired by recent success of feature enhancement methods in semi-supervised learning, we further introduce PROFORMER -an improved training strategy which uses soft-attention applied on iteratively estimated action category PROtotypes used to augment the embeddings and compute an auxiliary consistency loss. Extensive experiments consistently demonstrate the effectiveness of our approach for one-shot recognition from body poses, achieving state-of-the-art results on multiple datasets and surpassing the best published approach on the challenging NTU-120 one-shot benchmark by 1.84%. Our code will be made publicly available at https: //github.com/KPeng9510/ProFormer.

show abstract

HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm

Cited by 54 publications

References 38 publications

Spatial Reasoning and Context-Aware Attention Network for Skeleton-Based Action Recognition

Spatial Reasoning and Context-Aware Attention Network for Skeleton-Based Action Recognition

Learning 3D Skeletal Representation From Transformer for Action Recognition

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Contact Info

Product

Resources

About