Quo Vadis, Skeleton Action Recognition?

Abstract

In this paper, we study current and upcoming frontiers across the landscape of skeleton-based human action recognition. To begin with, we benchmark state-of-the-art models on the NTU-120 dataset and provide multi-layered assessment of the results. To examine skeleton action recognition 'in the wild', we introduce Skeletics-152, a curated and 3-D pose-annotated subset of RGB videos sourced from Kinetics-700, a large-scale action dataset. The results from benchmarking the top performers of NTU-120 on Skeletics-152 reveal the challenges and domain gap induced by actions 'in the wild'. We extend our study to include out-of-context actions by introducing Skeleton-Mimetics, a dataset derived from the recently introduced Mimetics dataset. Finally, as a new frontier for action recognition, we introduce Metaphorics, a dataset with caption-style annotated YouTube videos of the popular social game Dumb Charades and interpretative dance performances. Overall, our work characterizes the strengths and limitations of existing approaches and datasets. It also provides an assessment of top-performing approaches across a spectrum of activity settings and via the introduced datasets, proposes new frontiers for human action recognition.

PyHAPT: A Python-based Human Activity Pose Tracking data processing framework

“…It requires researchers to employ other pose estimation methods (e.g., OpenPose [17], OpenPifPaf [18], MMPose [19], VIBE [20]) to extract and pre-process the skeletal representation such that the pre-processed skeletal data could be ready for training and evaluating the deep learning recognition models. Gupta et al [21] made an effort to organize a couple of skeletal datasets obtained from other public datasets [22,23] that were still collected from constrained environments and crowd-sourcing methods rather than real public spaces (In The Wild-ITW ), except that they do not provide the software to elaborate these data. We analyzed some relevant skeleton-based HAR models since 2018 to check how public datasets were used to train and evaluate models in the community.…”

Rank-GCN for Robust Action Recognition

“…Skeletics-152: Skeletics-152 [1] is a skeleton action dataset extracted from the Kinetics700 [35] dataset with the VIBE [36] pose estimator. Because Kinetics-700 has some activities without people and some that are to classify within the context of what humans interact with, 152 classes out of the 700 total classes are chosen to build Skeletics-152.…”

“…According to the utilized types of input data, action recognition methods are roughly categorized into image-based, skeleton-based, and hybrid approaches. In image-based approaches, optical flows, which refer to the point correspondences across pairs of images have been commonly used to represent the apparent motions of subjects of interest [1]. However, these methods often require time-consuming and storage-demanding subprocesses.…”

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

“…Action recognition is fundamental in video-based tasks with many approaches proposed [20], [21], [22], [23], [24], [25], [26], [27], [28], [29] and datasets [30], [31], [18], [17], [19], [32], [33], [34]. We notice that there is also a trend for more fine-grained action understanding, from video classification [20], [21] to spatial-temporal action detection [32], [35], [36], [14], and human-part level action recognition [15].…”