Hierarchical Transformer: Unsupervised Representation Learning for Skeleton-Based Human Action Recognition

Cheng, Yi-Bin; Chen, Xipeng; Chen, Junhong; Wei, Pengxu; Zhang, Dongyu; Li, Lin

doi:10.1109/icme51207.2021.9428459

Cited by 38 publications

(23 citation statements)

References 20 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cheng et al [247] presented a hierarchical Transformer for unsupervised skeleton-based HAR, along with a motion prediction pre-training task between adjacent frames to learn discriminative representations. Liu et al [248] proposed a Kernel Attention Adaptive Graph Transformer Network (KA-AGTN), which is mainly composed of a skeleton graph transformer block to effectively capture the varying degrees of higher-order dependencies among joints, a temporal kernel attention module, and an adaptive graph strategy.…”

Section: Transformer-based Methodsmentioning

confidence: 99%

Human Action Recognition From Various Data Modalities: A Review

Sun

Rahmani

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

254

View full text Add to dashboard Cite

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

show abstract

Section: Transformer-based Methodsmentioning

confidence: 99%

Human Action Recognition From Various Data Modalities: A Review

Sun

Rahmani

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

254

View full text Add to dashboard Cite

show abstract

“…In the activity recognition area, Trear [41] proposes a transformer-based RGB-D egocentric activity recognition framework by adapting self-attention to model temporal structure from different modalities. Besides, action-transformer [42], motiontransformer [43], hierarchical-transformer [44], spatial temporal transformer network [45] and STST [46] are designed for skeleton-based activity recognition, modeling temporaland spatial dependencies in the skeleton sequences. MM-ViT [47] factorizes self-attention across the space, time, and modality dimensions, operating in the compressed video domain and exploiting various modalities.…”

Section: Related Workmentioning

confidence: 99%

TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Traditional video-based human activity recognition has experienced remarkable progress linked to the rise of deep learning, but this effect was slower as it comes to the downstream task of driver behavior understanding. Understanding the situation inside the vehicle cabin is essential for Advanced Driving Assistant System (ADAS) as it enables identifying distraction, predicting driver's intent and leads to more convenient human-vehicle interaction. At the same time, driver observation systems face substantial obstacles as they need to capture different granularities of driver states, while the complexity of such secondary activities grows with the rising automation and increased driver freedom. Furthermore, a model is rarely deployed under conditions identical to the ones in the training set, as sensor placements and types vary from vehicle to vehicle, constituting a substantial obstacle for real-life deployment of data-driven models. In this work, we present a novel vision-based framework for recognizing secondary driver behaviours based on visual transformers and an additional augmented feature distribution calibration module. This module operates in the latent feature-space enriching and diversifying the training set at feature-level in order to improve generalization to novel data appearances, (e.g., sensor changes) and general feature quality. Our framework consistently leads to better recognition rates, surpassing previous state-of-the-art results of the public Drive&Act benchmark on all granularity levels. Our code will be made publicly available at https: //github.com/KPeng9510/TransDARC.

show abstract

“…In NLP, masked language modelling is a pretraining technique where randomly masked tokens in the input are predicted. This approach has been explored for action recognition (Cheng et al 2021), where certain frames are masked and a regression task estimates coordinates of keypoints. In addition, a direction loss is also proposed to classify the quadrant where the motion vector lies.…”

Section: Masking-based Pretrainingmentioning

confidence: 99%

“…We follow the same hyperparameter settings as described in Motion-Transformer (Cheng et al 2021) Table 5: Effectiveness of pretraining strategies as measured on ISLR accuracy on INCLUDE random masking of 40% of the input frames. When using only the regression loss, we find that pretraining learns to reduce the loss as shown in Figure 3.…”

Section: Masking-based Pretrainingmentioning

confidence: 99%

OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Prem¹,

NC²,

Kumar³

et al. 2021

Preprint

View full text Add to dashboard Cite

AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sentences. We introduce OpenHands 1 , a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition. First, we propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference, and we release standardized pose datasets for 6 different sign languages -American, Argentinian, Chinese, Greek, Indian, and Turkish. Second, we train and release checkpoints of 4 posebased isolated sign language recognition models across all 6 languages, providing baselines and ready checkpoints for deployment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unlabelled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare different pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslingual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages more accessible.

show abstract

Hierarchical Transformer: Unsupervised Representation Learning for Skeleton-Based Human Action Recognition

Cited by 38 publications

References 20 publications

Human Action Recognition From Various Data Modalities: A Review

Human Action Recognition From Various Data Modalities: A Review

TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration

OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Contact Info

Product

Resources

About