Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization

Abstract

Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling either "actor-actor" or "actorcontext" relations. However, such direct first-order relations are not sufficient for localizing actions in complicated scenes. Some actors might be indirectly related via objects or background context in the scene. Such indirect relations are crucial for determining the action labels but are mostly ignored by existing work. In this paper, we propose to explicitly model the Actor-Context-Actor Relation, which can capture indirect high-order supportive information for effectively reasoning actors' actions in complex scenes. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel Highorder Relation Reasoning Operator to model indirect relations for spatiotemporal action localization. Moreover, to allow utilizing more temporal contexts, we extend our framework with an Actor-Context Feature Bank for reasoning long-range high-order relations. Extensive experiments on AVA dataset validate the effectiveness of our ACAR-Net. Ablation studies show advantages of modeling high-order relations over existing first-order relation reasoning methods. The proposed ACAR-Net is also the core module of our 1st place solution in AVA-Kinetics Crossover Challenge 2020. Training code and models will be available at https://github.com/Siyu-C/ACAR-Net.

Relation Modeling in Spatio-Temporal Action Localization

“…Maintaining feature memory bank for storing and utilizing representations along long term context has been demonstrated to be effective strategies for this task [25,20,16]. We also adapt the feature bank, which saves our pooled feature features and provides previously stored person features of timestamps within a long-range of current video clip.…”

“…Spatio-temporal action localization aims to localize atomic actions of people in videos with 3D bounding boxes, which has attract large efforts in recent years [5,25,20,16,4,9]. Generally, there are two main factors showing fundamental influence on the performance of this task, i.e.…”

“…The design of video networks has been widely studied [4,19,3] and greatly enhance the performance of downstream tasks. Besides, pretraining such networks on large-scale networks is also demonstrated to be effective [20,16], e.g. pretrain on Ki-netics700 [2].…”

“…For relation modeling, different approaches has been studied in the fields of computer vision [23,1], social networks [12,10] and nature language processing [22]. Specifically, transformed-based relation modeling has been proved for improving the spatio-temporal localization task [25,20,16].…”

Spatio-Temporal Context for Action Detection

“…Action detection is a task that consists in detecting people and recognizing their actions along videos. Being fundamental to video understanding, action detection has gained attention in recent years [5], [2], [6], leading to remarkable advances.…”