Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization

Abstract

Temporal action localization is an important yet challenging research topic due to its various applications. Since the frame-level or segment-level annotations of untrimmed videos require amounts of labor expenditure, studies on the weakly-supervised action detection have been springing up. However, most of existing frameworks rely on Class Activation Sequence (CAS) to localize actions by minimizing the video-level classification loss, which exploits the most discriminative parts of actions but ignores the minor regions. In this paper, we propose a novel weakly-supervised framework by adversarial learning of two modules for eliminating such demerits. Specifically, the first module is designed as a well-designed Seeded Sequence Growing (SSG) Network for progressively extending seed regions (namely the highly reliable regions initialized by a CASbased framework) to their expected boundaries. The second module is a specific classifier for mining trivial or incomplete action regions, which is trained on the shared features after erasing the seeded regions activated by SSG. In this way, a whole network composed of these two modules can be trained in an adversarial manner. The goal of the adversary is to mine features that are difficult for the action classifier. That is, erasion from SSG will force the classifier to discover minor or even new action regions on the input feature sequence, and the classifier will drive the seeds to grow, alternately. At last, we could obtain the action locations and categories from the well-trained SSG and the classifier. Extensive experiments on two public benchmarks THUMOS'14 and ActivityNet1.3 demonstrate the impressive performance of our proposed method compared with the state-of-the-arts.

FREE: A Fast and Robust End-to-End Video Text Spotter

“…With the rapid development of artificial intelligence techniques [18], [19], [20], [21], great progress has been made in many isolated applications such as causal inference [22], named entities identification [23], question answering [24], scene text spotting [5], [6], [17] and video understanding [25], [26]. However, it is very important to build multiple knowledge representation [27] for understanding the real and complex world.…”

Action Unit Memory Network for Weakly Supervised Temporal Action Localization

“…To handle the two problems, existing methods can be divided into three types. The first type of works attempt to solve the localization completeness by applying a well-designed erasing strategy [37,55,53] or a multi-branch architecture [21]. For example, Zhong et al [55] design a stepby-step erasion approach to train the one-by-one classifiers, via collecting detection results from these classifiers, more action segments are found.…”

“…To relieve this problem, the weakly supervised setting that only requires video-level category labels is proposed [37,55,39,37,55,53,29,30,45,46]. It can be formulated as a multiple instance learning problem, where a video is treated as a bag of multiple segments and fed into a video-level classifier to get a class activation sequence (CAS).…”

“…There are two primary challenges, named localization completeness and background interference. To solve the first challenge, previous works usually adopt a well-designed erasing strategy [37,55,53] or a multi-branch architecture [21]. Both of them aim to force the model to concentrate on different parts of videos and hence discover the whole action without missing any relevant segments.…”

Human-Aware Coarse-to-Fine Online Action Detection

“…The past decade has witnessed the great efforts in action understanding [1,2,3,4,5], among which action detection is receiving the most considerable attention [6,7,8,9,10,11,12]. Action detection targets predicting if an action occurs in a video that has its complete observation; meanwhile, finding the relevant spatial-temporal location.…”