Hierarchical Graph-Rnns for Action Detection of Multiple Activities

Biswas, Sovan; Souri, Yaser; Gall, Jüergen

doi:10.1109/icip.2019.8803650

Cited by 3 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we compare our approach with the state-of-the-art for fully supervised action detection in Table 4. Our approach is competitive to fully supervised approaches [5,6,7,8]. When we train our approach with full supervision, we improve over SlowFast [10] by +1.1% mAP on the validation set.…”

Section: Resultsmentioning

confidence: 93%

“…The graph connects all actors and we use a graph RNN to infer the action probabilities for each actor based on the spatial and temporal context. In our approach, we use the hierarchical Graph RNN (HGRNN) [7] where the features per node are obtained by ROI pooling over the 3D CNN feature maps. The HGRNN and 3D CNN are learned using the MIML loss (1).…”

Section: Actor-action Associationmentioning

confidence: 99%

“…In recent years, we have seen a major progress for spatially and temporally detecting actions in videos [1,2,3,4,5,6,7,8,9,10]. For this task, the bounding box of each person and their corresponding action labels need to be estimated for each frame as shown in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we move a step forward and introduce the task of weakly supervised multi-label spatio-temporal action detection with multiple actors in a video. The goal is to infer a list of multiple actions for each actor in a given video clip as in the fully supervised case [5,6,7,8,9,10]. However, in the weakly supervised setting only actions occurring in each training video are known.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting

Biswas¹,

Gall²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Since collecting and annotating data for spatio-temporal action detection is very expensive, there is a need to learn approaches with less supervision. Weakly supervised approaches do not require any bounding box annotations and can be trained only from labels that indicate whether an action occurs in a video clip. Current approaches, however, cannot handle the case when there are multiple persons in a video that perform multiple actions at the same time. In this work, we address this very challenging task for the first time. We propose a baseline based on multi-instance and multi-label learning. Furthermore, we propose a novel approach that uses sets of actions as representation instead of modeling individual action classes. Since computing the probabilities for the full power set becomes intractable as the number of action classes increases, we assign an action set to each detected person under the constraint that the assignment is consistent with the annotation of the video clip. We evaluate the proposed approach on the challenging AVA dataset where the proposed approach outperforms the MIML baseline and is competitive to fully supervised approaches.

show abstract