Pairwise Body-Part Attention for Recognizing Human-Object Interactions

Abstract

In human-object interactions (HOI) recognition, conventional methods consider the human body as a whole and pay a uniform attention to the entire body region. They ignore the fact that normally, human interacts with an object by using some parts of the body. In this paper, we argue that different body parts should be paid with different attention in HOI recognition, and the correlations between different body parts should be further considered. This is because our body parts always work collaboratively. We propose a new pairwise body-part attention model which can learn to focus on crucial parts, and their correlations for HOI recognition. A novel attention based feature selection method and a feature representation scheme that can capture pairwise correlations between body parts are introduced in the model. Our proposed approach achieved 10% relative improvement (36.1 mAP→ 39.9 mAP) over the state-of-the-art results in HOI recognition on the HICO dataset. We will make our model and source codes publicly available.

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

“…detection. Different from HOI recognition [7,6,4,5,20] which is an image level classification problem, HOI detection needs to detect interactive human-object pairs and classify their interactions at instance level. With the assistance of DNNs and large-scale datasets, recently methods have made significant progress.…”

RGB-D salient object detection: A survey

“…Attention mechanisms have been widely applied to various deep learning-based tasks [178][179][180][181], allowing networks to selectively pay attention to a subset of regions for extracting powerful and discriminative features. Co-attention mechanisms have also been developed to explore the underlying correlations between multiple modalities.…”

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

“…2D human-object interaction. Perceiving human-object interactions in 2D images has been studied extensively [5,9,10,15,16,22,23,24,35,39,40,43]. Gkioxari et al [10] detect (human, verb, object) triplets using human appearance as cues to localize interacted objects.…”

“…Gkioxari et al [10] detect (human, verb, object) triplets using human appearance as cues to localize interacted objects. Fang et al [9] learn a pairwise body-part attention model, which focuses on crucial body parts and their corresponding interactions. Wang et al [39] predict interaction points to localize and classify the interaction directly.…”