Learning discriminative space-time actions from weakly labelled videos

Abstract

Current state-of-the-art action classification methods extract feature representations from the entire video clip in which the action unfolds, however this representation may include irrelevant scene context and movements which are shared amongst multiple action classes. For example, a waving action may be performed whilst walking, however if the walking movement and scene context appear in other action classes, then they should not be included in a waving movement classifier. In this work, we propose an action classification framework in which more discriminative action subvolumes are learned in a weakly supervised setting, owing to the difficulty of manually labelling massive video datasets. The learned models are used to simultaneously classify video clips and to localise actions to a given space-time subvolume. Each subvolume is cast as a bag-offeatures (BoF) instance in a multiple-instance-learning framework, which in turn is used to learn its class membership. We demonstrate quantitatively that even with single fixedsized subvolumes, the classification performance of our proposed algorithm is superior to the state-of-the-art BoF baseline on the majority of performance measures, and shows promise for space-time action localisation on the most challenging video datasets.

Action Recognition Using Discriminative Structured Trajectory Groups

“…A video is represented as a bag of kinematic modes. Closely related to our work is the work of Sapienza et al [16] where discriminative action subvolumes are learned in a weakly supervised setting. The learned models are used to classify and localize actions.…”

“…Instead of using subvolumes representation, we use trajectories and extract trajectory groups [14] from the video and aim to learn the discriminative trajectory groups to represent the video. Most importantly, our representation maintains the structural spatio-temporal information in each bag, while [16] treated each instances in a bag independently.…”

Person Re-identification by Video Ranking

“…Moreover, discriminative space-time video patches have also been exploited for action recognition [37]. However, action recognition approaches are Person re-identification challenges in public space scenes [42].…”

Hollywood 3D: What are the Best 3D Features for Action Recognition?

“…Laptev et alattempt to mitigate this by splitting the spatio-temporal volume into sub-blocks, creating a descriptor for each sub-block, and concatenating them to create the sequence descriptor (Laptev et al 2008). Sapienza et alfollow a similar vein, encoding individual sub-sequences, however rather than concatenating to create a single descriptor, they employ Multiple Instance Learning (MIL) (Sapienza et al 2012). This accounts for some parts of the sequence being irrelevant, for example before and after the action.…”