TSA-Net: Tube Self-Attention Network for Action Quality Assessment

Abstract

In recent years, assessing action quality from videos has attracted growing attention in computer vision community and humancomputer interaction. Most existing approaches usually tackle this problem by directly migrating the model from action recognition tasks, which ignores the intrinsic differences within the feature map such as foreground and background information. To address this issue, we propose a Tube Self-Attention Network (TSA-Net) for action quality assessment (AQA). Specifically, we introduce a single object tracker into AQA and propose the Tube Self-Attention Module (TSA), which can efficiently generate rich spatio-temporal contextual information by adopting sparse feature interactions. The TSA module is embedded in existing video networks to form TSA-Net. Overall, our TSA-Net is with the following merits: 1) High computational efficiency, 2) High flexibility, and 3) The state-of-theart performance. Extensive experiments are conducted on popular action quality assessment datasets including AQA-7 and MTL-AQA. Besides, a dataset named Fall Recognition in Figure Skating (FR-FS) is proposed to explore the basic action assessment in the figure skating scene. Our TSA-Net achieves the Spearman's Rank Correlation of 0.8476 and 0.9393 on AQA-7 and MTL-AQA, respectively, which are the new state-of-the-art results. The results on FR-FS also verify the effectiveness of the TSA-Net. The code and FR-FS dataset are publicly available at https:// github.com/ Shunli-Wang/ TSA-Net.

Action Quality Assessment with Temporal Parsing Transformer

“…Under 'w/o DD', out method achieves 0.9451(Sp. Corr) and 0.3222(R-ℓ 2 ), outperforms the CoRe and recent TSA-Net [27]. It's worth noting that TSA-Net utilizes an external VOT tracker [35] to extract human locations and then enhance backbone features, which is orthogonal to the main issue of temporal parsing addressed in our work.…”

“…On MTL-AQA dataset, we evaluated our experiments with two different settings, following prior work [32]. Specifically, the MTL-AQA dataset contains Pose+DCT [20] 0.2682 -C3D-SVR [19] 0.7716 -C3D-LSTM [19] 0.8489 -MSCADC-STL [18] 0.8472 -C3D-AVG-STL [18] 0.8960 -MSCADC-MTL [18] 0.8612 -C3D-AVG-MTL [18] 0.9044 -USDL [22] 0.9066 0.654 CoRe [32] 0.9341 0.365 TSA-Net [27] 0.9422 -Ours 0.9451 0.3222 Method (w/ DD) Sp. Corr R-ℓ2(× 100) USDL [22] 0.9231 0.468 MUSDL [22] 0.9273 0.451 CoRe [32] 0.9512 0.260 Ours 0.9607 0.2378 the label of difficult degree, and each video's quality score is calculated by the multiplication of the raw score with its difficulty.…”

A Survey of Video-based Action Quality Assessment

“…Fall Recognition in Figure Skating (FR-FS) [11]. Existing AQA datasets of figure skating [5], [10] only contain long videos, which will lead to the inundation of detailed information, and finally affect the evaluation performance of the AQA model.…”

“…Existing AQA datasets of figure skating [5], [10] only contain long videos, which will lead to the inundation of detailed information, and finally affect the evaluation performance of the AQA model. To solve this problem, Wang et al [11] proposed FR-FS to recognize figure skating falls and planned to build a delicate granularity AQA system gradually. Videos in FR-FS contains the movements of the athlete's take-off, rotation, and landing.…”

“…Action units are encoded from dense trajectories with a LSTM network. Wang et al [11] proposed a feature aggregation mechanism named tube self-attention module to efficiently generate rich spatio-temporal contextual information by adopting sparse feature interactions. Some studies focus on the design of network loss to obtain better action quality assessment performance.…”