One-shot action recognition in challenging therapy scenarios

Sabater, Alberto; Santos, Laura; Santos-Victor, José; Bernardino, Alexandre; Montesano, Luis; Murillo, Ana C.

doi:10.1109/cvprw53098.2021.00312

Cited by 24 publications

(11 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…III-C (the last group of approaches in Table I). The results further validated through three-shot experiments [39] 42.1 ---AP † [38] 42.9 ---APSR [26] 45.3 ---TCN-OneShot [55] 46.3 ---SL-DML [15] 50.9 ---Skeleton-DML [40] 54. ProFormer SL-DML Fig.…”

Section: B Experiments Resultsmentioning

confidence: 73%

“…We perform comprehensive studies for one-shot human activity recognition from 3D body poses on three challenging datasets: NTU-60 [54], NTU-120 [26] and Toyota Smarthome [30]. To compare with competitive state-of-the art methods for data-efficient recognition, we select NTU-120 as our primary test bed, as it is a well-established benchmark for one-shot recognition from 3D body poses [15], [26], [38], [39], [55]. Additionally, we adapt the evaluation protocols of Toyota Smarthome and NTU-60 to suit our data-scarce representation learning task.…”

Section: Experiments a Datasets And Implementation Detailsmentioning

confidence: 99%

See 1 more Smart Citation

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatically understanding human behaviour allows household robots to identify the most critical needs and plan how to assist the human according to the current situation. However, the majority of such methods are developed under the assumption that a large amount of labelled training examples is available for all concepts-of-interest. Robots, on the other hand, operate in constantly changing unstructured environments, and need to adapt to novel action categories from very few samples. Methods for data-efficient recognition from body poses increasingly leverage skeleton sequences structured as image-like arrays and then used as input to convolutional neural networks. We look at this paradigm from the perspective of transformer networks, for the first time exploring visual transformers as data-efficient encoders of skeleton movement. In our pipeline, body pose sequences cast as image-like representations are converted into patch embeddings and then passed to a visual transformer backbone optimized with deep metric learning. Inspired by recent success of feature enhancement methods in semi-supervised learning, we further introduce PROFORMER -an improved training strategy which uses soft-attention applied on iteratively estimated action category PROtotypes used to augment the embeddings and compute an auxiliary consistency loss. Extensive experiments consistently demonstrate the effectiveness of our approach for one-shot recognition from body poses, achieving state-of-the-art results on multiple datasets and surpassing the best published approach on the challenging NTU-120 one-shot benchmark by 1.84%. Our code will be made publicly available at https: //github.com/KPeng9510/ProFormer.

show abstract

Section: B Experiments Resultsmentioning

confidence: 73%

Section: Experiments a Datasets And Implementation Detailsmentioning

confidence: 99%

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…After GCN module extracts movement representation features for all nodes, we deploy a TCN module to learn features in temporal domain [40]. Before entering TCN module, we need to participate the whole sequence in multiple segments according to the temporal resolution for anomaly state locating task.…”

Section: A Behavior Feature Extractingmentioning

confidence: 99%

OBTAIN: Observational Therapy-Assistance Neural Network for Training State Recognition

Wang

Yang

2023

IEEE Access

View full text Add to dashboard Cite

Children with autism spectrum disorder (ASD) often require long-term and highquality intervention. An important factor affecting the quality of the intervention is therapists' observation skill, by which therapists can opportunely adjust their intervention strategies based on children's states. However, there is a shortage of experienced therapists and observational skill development is time-consuming for junior therapists to acquire. This motivates us to use data-driven deep learning method to build an OBservational Therapy-AssIstance Neural network (OBTAIN), which is a weakly-supervised learning framework for the therapy training states recognition. OBTAIN first represents children's skeleton-sequence data as a large graph. Then a graph representation learning module is used to extract training state features. To learn spatialtemporal behavior features more effectively and efficiently, a novel structure-aware GCN block is designed in OBTAIN's learning module. Finally, a MILnet and corresponding joint MIL loss are used to learn state score prediction from extracted features. Experimental results (0.824 AUC score) on DREAM dataset demonstrate our OBTAIN can effectively recognize the training states in autism intervention.

show abstract

“…The capability to understand human intentions and goals allows a robot to discern when its assistance is most needed, thereby minimizing disruptions to human activities. A robot equipped with a human action recognition system can also be used to monitor the condition of patients to provide better daily-life assistance for their recovery [6], [7], assess the safety of its surroundings, issue warnings, and detect gestures for help in rescue missions to provide assistance. Challenges of image-or video-based This work was supported in part by the SmartAge project sponsored by the Carl Zeiss Stiftung (P2019-01-003; 2021-2026), the MWK through the Cooperative Graduate School Accessibility through AI-based Assistive Technology (KATE) under Grant BW6-03, and in part by the BMBF through a fellowship within the IFI program of the German Academic Exchange Service (DAAD), in part by the HoreKA@KIT supercomputer partition, and in part by Hangzhou SurImage Technology Company Ltd. *Corresponding author.…”

Section: Introductionmentioning

confidence: 99%

Delving Deep Into One-Shot Skeleton-Based Action Recognition With Diverse Occlusions

Peng

Roitberg

Yang

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

To integrate action recognition methods into autonomous robotic systems, it is crucial to consider adverse situations involving target occlusions. Such a scenario, despite its practical relevance, is rarely addressed in existing selfsupervised skeleton-based action recognition methods. To empower robots with the capacity to address occlusion, we propose a simple and effective method. We first pre-train using occluded skeleton sequences, then use k-means clustering (KMeans) on sequence embeddings to group semantically similar samples. Next, we employ K-nearest-neighbor (KNN) to fill in missing skeleton data based on the closest sample neighbors. Imputing incomplete skeleton sequences to create relatively complete sequences as input provides significant benefits to existing skeleton-based self-supervised models. Meanwhile, building on the state-of-the-art Partial Spatio-Temporal Learning (PSTL), we introduce an Occluded Partial Spatio-Temporal Learning (OPSTL) framework. This enhancement utilizes Adaptive Spatial Masking (ASM) for better use of high-quality, intact skeletons. The effectiveness of our imputation methods is verified on the challenging occluded versions of the NTURGB+D 60 and NTURGB+D 120. The source code will be made publicly available at https://github.com/cyfml/OPSTL.

show abstract

One-shot action recognition in challenging therapy scenarios

Cited by 24 publications

References 18 publications

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

OBTAIN: Observational Therapy-Assistance Neural Network for Training State Recognition

Delving Deep Into One-Shot Skeleton-Based Action Recognition With Diverse Occlusions

Contact Info

Product

Resources

About