Revisiting spatio-temporal layouts for compositional action recognition

Abstract

Recognizing human actions is fundamentally a spatio-temporal reasoning problem, and should be, at least to some extent, invariant to the appearance of the human and the objects involved. Motivated by this hypothesis, in this work, we take an object-centric approach to action recognition. Multiple works have studied this setting before, yet it remains unclear (i) how well a carefully crafted, spatio-temporal layout-based method can recognize human actions, and (ii) how, and when, to fuse the information from layoutand appearance-based models. The main focus of this paper is compositional/few-shot action recognition, where we advocate the usage of multi-head attention (proven to be effective for spatial reasoning) over spatio-temporal layouts, i.e., configurations of object bounding boxes. We evaluate different schemes to inject video appearance information to the system, and benchmark our approach on background cluttered action recognition. On the Something-Else and Action Genome datasets, we demonstrate (i) how to extend multi-head attention for spatio-temporal layout-based action recognition, (ii) how to improve the performance of appearance-based models by fusion with layout-based models, (iii) that even on non-compositional background-cluttered video datasets, a fusion between layout-and appearance-based models improves the performance.

Hand-Object Interaction Reasoning

“…Obj. Top-1 Top-5 STIN [26] 37.2 62.4 I3D+STIN [26] 48.2 72.6 CAF [30] 52.3 78.9 STRG [42] 52.3 78.3 IRN (Ours) 52.9 80.8 I3D-STIN [26] 51.5 77.1 STRG-STIN [26] 56.2 81.3 CACNF [30] 56.9 82.5 224 × 224. We refer to this augmentation approach as SCR, which we use in all our experiments on EPIC-KITCHENS-100.…”

Heterogeneous Feature Fusion for Improving Performance of Action Detection

“…Such information can be readily and reliably extracted by modern deep learning algorithms and have been reported to enhance the accuracy of action recognition [6]. For example, [7] exploited the positional relations between instances and object categories, and achieved accurate scene-level object-centered action recognition. Recently, [8] proposed a new bi-modal network for action detection, which has an RGB stream and a pose stream, and demonstrated that the heterogeneous features provide essential information for accurate action detection.…”