Action Quality Assessment with Temporal Parsing Transformer

Abstract

Action Quality Assessment(AQA) is important for action understanding and resolving the task poses unique challenges due to subtle visual differences. Existing state-of-the-art methods typically rely on the holistic video representations for score regression or ranking, which limits the generalization to capture fine-grained intra-class variation. To overcome the above limitation, we propose a temporal parsing transformer to decompose the holistic feature into temporal part-level representations. Specifically, we utilize a set of learnable queries to represent the atomic temporal patterns for a specific action. Our decoding process converts the frame representations to a fixed number of temporally ordered part representations. To obtain the quality score, we adopt the state-of-the-art contrastive regression based on the part representations. Since existing AQA datasets do not provide temporal part-level labels or partitions, we propose two novel loss functions on the cross attention responses of the decoder: a ranking loss to ensure the learnable queries to satisfy the temporal order in cross attention and a sparsity loss to encourage the part representations to be more discriminative. Extensive experiments show that our proposed method outperforms prior work on three public AQA benchmarks by a considerable margin.

A Video-Based Augmented Reality System for Human-in-the-Loop Muscle Strength Assessment of Juvenile Dermatomyositis

“…This allows us to assess muscle strength across all actions in a comprehensive manner. (2) The corresponding character for a given action is selected from the previously generated animation dataset, which does not contain any personalized design. Therefore, it is possible to generate more realistic motions by combining the motionstyle transfer technique.…”

“…Different classification criteria can be used to categorize these AQA methods. As for the input format, these frameworks include exemplarbased [2,46,48] and exemplar-free [37,42] methods depending on whether exemplars are used or not. The former usually involves selecting a set of exemplars along with the target sample as input, while the latter does not.…”

“…The former usually involves selecting a set of exemplars along with the target sample as input, while the latter does not. For example, Bai et al [2] learned the relative score of the input action relative to the exemplar, and used the exemplar's score to calculate the final quality score. Compared to exemplar-free methods, this method is easier to regress relative scores.…”

“…It is easily over-fitting when training large-sized models. Previous works [2,45,48] have hypothesized that the action recognition features can also be applied to the AQA task. So we choose I3D [5] pre-trained on the Kinetcis-400 dataset [18] as the feature extractor.…”

“…Existing AQA methods [2,48] ignore the intrinsic heterogeneity among the feature spaces of action recognition and AQA, i.e., the model pretrained on action recognition datasets can be sub-optimal for AQA. To compensate for this, we first design feature distance loss to regularize the heterogeneous feature space:…”