Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition

Men, Qianhui; Ho, Edmond S. L.; Shum, Hubert P. H.; Leung, Howard

doi:10.1016/j.neucom.2023.03.070

Cited by 8 publications

(8 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These contrastive losses have been widely used in recent selfsupervised action recognition methods. For example, CrosS-CLR [16] and SkeletonMixCLR [17] achieve promising performance by adopting MoCov2 [41]. AimCLR [15] adopts InfoNCE [22] as a contrastive loss to improve the model performance on downstream tasks.…”

Section: Related Work a Self-supervised Skeleton-based Action Recogni...mentioning

confidence: 99%

“…For example, MCC [20] adopts a speed-changed operation combined with a random start frame to build hard positive samples. CrosSCLR [16] adopts Shear and Crop data augmentation, which has become the most commonly used skeleton data augmentation strategies in self-supervised action recognition works [16], [17], [15].…”

Section: B Skeleton Data Augmentation Strategymentioning

confidence: 99%

“…AimCLR [15] adopts more hand-crafted data augmentation approaches, such as Spatial Flipping, Rotation, Axis Masking, Temporal Flipping, to construct hard contrastive pairs. Furthermore, SkeletonMixCLR [17] develops a stronger skeleton data augmentation method by randomly mixing different body parts (e.g., left hands, right legs, trunk) among different skeleton sequences. Although these hand-crafted data augmentation methods can create effective contrastive pairs, their random data augmentation strategies may construct sufficient challenging contrastive pairs.…”

Section: B Skeleton Data Augmentation Strategymentioning

confidence: 99%

“…However, fully-supervised methods heavily rely on a large amount of annotated skeleton data, but annotating data is high-cost and time-consuming. To complement the lack of annotated skeleton data, some recent action recognition works [12], [13], [14], [15], [16], [17] have delved into exploring self-supervised approaches.…”

Section: Introductionmentioning

confidence: 99%

“…In the self-supervised skeleton-based action recognition task, contrastive learning [18], [19] has been introduced to learn feature representations. Those self-supervised action recognition methods [15], [16], [20], [17] first augment skeleton data to construct contrastive pairs (i.e., positive and negative pairs). Then, they learn action features by increasing the similarity between positive pairs while decreasing the similarity between negative pairs using a contrastive loss [21], [22], [23].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition

Guan¹,

Yu²,

Huang³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this work, we propose a new Dual Min-Max Games (DMMG) based self-supervised skeleton action recognition method by augmenting unlabeled data in a contrastive learning framework. Our DMMG consists of a viewpoint variation min-max game and an edge perturbation min-max game. These two min-max games adopt an adversarial paradigm to perform data augmentation on the skeleton sequences and graphstructured body joints, respectively. Our viewpoint variation min-max game focuses on constructing various hard contrastive pairs by generating skeleton sequences from various viewpoints. These hard contrastive pairs help our model learn representative action features, thus facilitating model transfer to downstream tasks. Moreover, our edge perturbation min-max game specializes in building diverse hard contrastive samples through perturbing connectivity strength among graph-based body joints. The connectivity-strength varying contrastive pairs enable the model to capture minimal sufficient information of different actions, such as representative gestures for an action while preventing the model from overfitting. By fully exploiting the proposed DMMG, we can generate sufficient challenging contrastive pairs and thus achieve discriminative action feature representations from unlabeled skeleton data in a self-supervised manner. Extensive experiments demonstrate that our method achieves superior results under various evaluation protocols on widely-used NTU-RGB+D and NTU120-RGB+D datasets.

show abstract