Multi-level Sequence GAN for Group Activity Recognition

Abstract

We propose a novel semi supervised, Multi Level Sequential Generative Adversarial Network (MLS-GAN ) architecture for group activity recognition. In contrast to previous works which utilise manually annotated individual human action predictions, we allow the models to learn it's own internal representations to discover pertinent subactivities that aid the final group activity recognition task. The generator is fed with person-level and scene-level features that are mapped temporally through LSTM networks. Action-based feature fusion is performed through novel gated fusion units that are able to consider long-term dependancies, exploring the relationships among all individual actions, to learn an intermediate representation or 'action code' for the current group activity. The network achieves it's semi-supervised behaviour by allowing it to perform group action classification together with the adversarial real/fake validation. We perform extensive evaluations on different architectural variants to demonstrate the importance of the proposed architecture. Furthermore, we show that utilising both person-level and scene-level features facilitates the group activity prediction better than using only person-level features. Our proposed architecture outperforms current state-of-the-art results for sports and pedestrian based classification tasks on Volleyball and Collective Activity datasets, showing it's flexible nature for effective learning of group activities. 1

A Comprehensive Review of Group Activity Recognition in Videos

“…Sometimes, different group activities share the same local motion which may cause misclassifications. To reduce the influence of confused motions, Kim et al [73] proposed a discriminative group context feature (DGCF) that takes prominent sub-events into consideration. Two types of features, individual activity and sub-event feature, are extracted to construct group activity representations.…”

Temporarily-Aware Context Modeling Using Generative Adversarial Networks for Speech Activity Detection

“…We exploit the task-specific loss-function learning capability of the GAN framework to automatically learn a custom loss function [30]- [33] that facilitates these two tasks . The merit of this approach is that it allows us to learn a highly nonlinear loss, in contrast to a linear loss like cross entropy, to optimally capture the underlying semantics of the process.…”

“…This custom loss function learning capability of GANs is highly beneficial in the multi-task learning setting, as it allows us to learn a custom loss function that accounts for all the tasks at hand rather than simply adding together the loss functions for individual tasks. For instance, in [33] the authors illustrate the utility of GANs for video based action prediction while synthesising future frame representations, and the authors in [34] showed that this process is highly beneficial for mitigating the errors due to variation of view angles in gait recognition through view synthesis.…”

Predicting the Future: A Jointly Learnt Model for Action Anticipation

“…In our work we utilise a conditional GAN [12,13,36] for deep future representation generation. A limited number The model receives RGB and optical flow streams as the visual and temporal representations of the given scene.…”