Action Transformer: A self-attention model for short-time pose-based human action recognition

Mazzia, Vittorio; Angarano, Simone; Salvetti, Francesco; Angelini, Federico; Chiaberge, Marcello

doi:10.1016/j.patcog.2021.108487

Cited by 138 publications

(32 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related work usually differentiates human activities based on the target task or application. Simple atomic actions, poses and motions are usually easy to identify accurately with various machine learning models [1,9,11,13], whereas composite activities, such as ADLs require either more intricate models or a more complex feature engineering [16]. It is a usual approach to decompose complex ADLs into simpler activities, which are generally easier to recognize [24].…”

Section: Related Workmentioning

confidence: 99%

Transformer-Based Recognition of Activities of Daily Living from Wearable Sensor Data

Augustinov

NISAR²,

et al. 2022

Preprint

View full text Add to dashboard Cite

Smart support systems for the recognition of Activities of Daily Living (ADLs) can help elderly people live independently for longer improving their standard of living. Many machine learning approaches have been proposed lately for Human Activity Recognition (HAR), including elaborated networks that contain convolutional, recurrent, and attentive layers. The ubiquity of wearable devices has provided an increasing amount of time-series data that can be used for such applications in an unobtrusive manner. But there are not many studies on the performance of the attention-based Transformer model in HAR, especially not for complex activities such as ADLs. This work implements and evaluates the novel self-attention Transformer model for the classification of ADLs and compares it to the already well-established approach of recurrent Long-Short Term Memory (LSTM) networks. The proposed method is a two-level hierarchical model, in which atomic activities are initially recognized in the first step and their probability scores are extracted and utilized for the Transformer-based classification of seven more complex ADLs in the second step. The Transformer is used at the second step to classify seven ADLs. Our results show that the Transformer model reaches the same performance and even outperforms LSTM networks cleary in the subject-dependent configuration (73.36 % and 69.09 %), while relying only on attention-mechanism to depict global dependencies between input and output without the need to use any recurrence. The proposed model was tested using two different segment lengths, indicating its effectiveness in learning long-range dependencies of shorter actions in complex activities.

show abstract

Section: Related Workmentioning

confidence: 99%

Transformer-Based Recognition of Activities of Daily Living from Wearable Sensor Data

Augustinov

NISAR²,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In the activity recognition area, Trear [41] proposes a transformer-based RGB-D egocentric activity recognition framework by adapting self-attention to model temporal structure from different modalities. Besides, action-transformer [42], motiontransformer [43], hierarchical-transformer [44], spatial temporal transformer network [45] and STST [46] are designed for skeleton-based activity recognition, modeling temporaland spatial dependencies in the skeleton sequences. MM-ViT [47] factorizes self-attention across the space, time, and modality dimensions, operating in the compressed video domain and exploiting various modalities.…”

Section: Related Workmentioning

confidence: 99%

TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Traditional video-based human activity recognition has experienced remarkable progress linked to the rise of deep learning, but this effect was slower as it comes to the downstream task of driver behavior understanding. Understanding the situation inside the vehicle cabin is essential for Advanced Driving Assistant System (ADAS) as it enables identifying distraction, predicting driver's intent and leads to more convenient human-vehicle interaction. At the same time, driver observation systems face substantial obstacles as they need to capture different granularities of driver states, while the complexity of such secondary activities grows with the rising automation and increased driver freedom. Furthermore, a model is rarely deployed under conditions identical to the ones in the training set, as sensor placements and types vary from vehicle to vehicle, constituting a substantial obstacle for real-life deployment of data-driven models. In this work, we present a novel vision-based framework for recognizing secondary driver behaviours based on visual transformers and an additional augmented feature distribution calibration module. This module operates in the latent feature-space enriching and diversifying the training set at feature-level in order to improve generalization to novel data appearances, (e.g., sensor changes) and general feature quality. Our framework consistently leads to better recognition rates, surpassing previous state-of-the-art results of the public Drive&Act benchmark on all granularity levels. Our code will be made publicly available at https: //github.com/KPeng9510/TransDARC.

show abstract

“…Bai et al [20] introduced an approach based on multi-range feature interchange to capture short-range motion features and long-range dependencies. Finally, advanced transformer-based approaches have been proposed for action recognition [21], [5], [22], [23], [24], [25], often exploiting skeleton points [26], [27], [28], [29], [30]. Different from our work, these approaches consider a standard supervised setting, where all data is labelled and there is no domain shift between training and evaluation datasets.…”

Section: Introductionmentioning

confidence: 99%

Dual-Head Contrastive Domain Adaptation for Video Action Recognition

Costa

Zara

Rota

et al. 2022

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Over the last few years, Unsupervised Domain Adaptation (UDA) techniques have acquired remarkable importance and popularity in computer vision. However, when compared to the extensive literature available for images, the field of videos is still relatively unexplored. On the other hand, the performance of a model in action recognition is heavily affected by domain shift. In this paper, we propose a simple and novel UDA approach for video action recognition. Our approach leverages recent advances on spatio-temporal transformers to build a robust source model that better generalises to the target domain. Furthermore, our architecture learns domain invariant features thanks to the introduction of a novel alignment loss term derived from the Information Bottleneck principle. We report results on two video action recognition benchmarks for UDA, showing state-of-the-art performance on HMDB↔UCF, as well as on Kinetics→NEC-Drone, which is more challenging. This demonstrates the effectiveness of our method in handling different levels of domain shift. The source code is available at https://github.com/vturrisi/UDAVT.

show abstract

Action Transformer: A self-attention model for short-time pose-based human action recognition

Cited by 138 publications

References 21 publications

Transformer-Based Recognition of Activities of Daily Living from Wearable Sensor Data

Transformer-Based Recognition of Activities of Daily Living from Wearable Sensor Data

TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration

Dual-Head Contrastive Domain Adaptation for Video Action Recognition

Contact Info

Product

Resources

About