Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos

Abstract

The state-of-the art solutions for human activity understanding from a video stream formulate the task as a spatio-temporal problem which requires joint localization of all individuals in the scene and classification of their actions or group activity over time. Who is interacting with whom, e.g. not everyone in a queue is interacting with each other, is often not predicted. There are scenarios where people are best to be split into sub-groups, which we call social groups, and each social group may be engaged in a different social activity. In this paper, we solve the problem of simultaneously grouping people by their social interactions, predicting their individual actions and the social activity of each social group, which we call the social task. Our main contributions are: i) we propose an end-to-end trainable framework for the social task; ii) our proposed method also sets the state-of-the-art results on two widely adopted benchmarks for the traditional group activity recognition task (assuming individuals of the scene form a single group and predicting a single group activity label for the scene); iii) we introduce new annotations on an existing group activity dataset, re-purposing it for the social task. The data and code for our method is publicly available 4 .

A Comprehensive Review of Group Activity Recognition in Videos

“…Inspired by a transformer network [80] which relies on self-attention mechanisms to allow the network to adaptively extract the most relevant information and relationships, Gavrilyuk et al [21] proposed an actor-transformers network which learns interactions between the actors and adaptively extracts the important information for activity recognition.…”

COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality

“…Several works tackle this problem from a graphbased perspective [40,63,100,101] such as applying Graph Convolutional Networks (GCNs) [49,96]. More recent works utilize attention modeling [63,73,98,103] including using Transformers [26,57] with a focus on determining the most critical persons [26,72,96,103], groups [24,57], or interactions [101]. Existing works have primarily use RGB-and optical-flow-based features with RoIAlign [33] to represent individuals [6,73,96,100].…”

“…COMPOSER facilitates compositional learning and a high-level semantic understanding of the video with a Multiscale Transformer that performs relational reasoning over these tokens scale by scale. and crowd behavior analysis [24,26,72,104]. Compared to the single-person atomic action recognition task, GAR requires addressing two additional challenges.…”

“…Existing work has proposed to jointly learn the group activity with individual actions [3,6,39,41,73,79] or person sub-groups [24,57,67] for a compositional understanding of the group activity. Meanwhile, graph [36,40,96,104] and transformer [26,57] based models have been proposed for relational reasoning over scene entities.…”

“…However, these methods fail to elaborately design compositional learning for GAR. They do not sufficiently make use of the multiscale scene elements in the GAR task by modeling over entities at either one semantic scale (e.g., person [26,36,96,104]) or two scales (e.g., person and subgroup [24,57,67]), or keypoint and person [71]). In addition, explicit multiscale modeling is neglected, lacking consistent compositional representations for the group action tasks.…”