Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning

Abstract

Weakly-supervised action localization problem requires training a model to localize the action segments in the video given only video level action label. It can be solved under the Multiple Instance Learning (MIL) framework, where a bag (video) contains multiple instances (action segments). Since only the bag's label is known, the main challenge is to assign which key instances within the bag trigger the bag's label. Most previous models use an attention-based approach. These models use attention to generate bag's representation from instances and then train it via bag's classification. In this work, we explicitly model the key instances assignment as a hidden variable and adopt an Expectation-Maximization framework. We derive two pseudo-label generation schemes to model the E and M process and iteratively optimize the likelihood lower bound. We also show that previous attention-based models implicitly violate the MIL assumptions that instances in negative bags should be uniformly negative. In comparison, Our EM-MIL approach more accurately models these assumptions. Our model achieves state-of-the-art performance on two standard benchmarks, THUMOS14 and ActivityNet1.2, and shows the superiority of detecting relative complete action boundary in videos containing multiple actions.

Deep Learning-based Action Detection in Untrimmed Videos: A Survey

“…DGAM [116] addressed the actioncontext confusion through imposing different attentions on different features with a generative model. EM-MIL [165] employed Expectation-Maximization to capture complete action instances and outperformed DGAM [116] on THUMOS14 dataset. Direct Action Localization.…”

“…DGAM [116] EM-MIL [165] + Conditional VAE / Expectation-Maximization to separate actions from context frames and capture complete action instances. -Not modeling temporal dependencies and relation between sub-actions.…”

Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization

“…There are some methods have noticed the importance of foreground-action consistency. For example, Refine-Loc [29] generated snippet-level hard pseudo labels by expanding previous detection results, TSCN [47] generated pseudo ground truth from the foreground attention sequence, and EM-MIL [21] put the pseudo-label generation into an expectation-maximization framework. There are also some methods [27,24,10] attempted to use class activations as a top-down supervision to guide foreground attention generation.…”

Action Unit Memory Network for Weakly Supervised Temporal Action Localization

“…For example, to decrease the intra-class variance, 3C-Net [28] and A2CL-PT [27] maintain a set of class center and RPN [12] learns class-specific prototypes. The third type of works are based on a class-agnostic attention mechanism [29,30,16,34,10,52,25], which can consider both the challenges simultaneously. Unlike the second type of works, the attention here is generated in a bottom-up way from the raw data and trained for highlighting foreground segments.…”

“…And based on the observation that background features differ from action features, DGAM [34] adopts a conditional variation auto-encoder to construct different feature distributions conditioned on the attention. Recently, TSCN [52] and EM-MIL [25] fuse the output of different modalities (RGB and optical flow) to generate pseudo labels for guidance of the attention.…”