Detecting Human-Object Interactions via Functional Generalization

Bansal, Ankan; Rambhatla, Sai Saketh; Shrivastava, Abhinav; Chellappa, Rama

doi:10.48550/arxiv.1904.03181

Cited by 4 publications

(15 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, different from [26] who aims to zero-shot learning, VCL targets at Generalized Zero-Shot Learning [34]. In [3], a generic object detector was incorporated to generalize to interactions involving previously unseen objects. Also, Yang et al [37] proposed to alleviate the predicate bias to objects for zero-shot visual relationship detection.…”

Section: Low-shot and Zero-shot Learningmentioning

confidence: 99%

See 1 more Smart Citation

Visual Compositional Learning for Human-Object Interaction Detection

Huang

Peng

Qiao

et al. 2020

Preprint

View full text Add to dashboard Cite

Human-Object interaction (HOI) detection aims to localize and infer relationships between human and objects in an image. It is challenging because an enormous number of possible combinations of objects and verbs types forms a long-tail distribution. We devise a deep Visual Compositional Learning (VCL) framework, which is a simple yet efficient framework to effectively address this problem. VCL first decomposes an HOI representation into object and verb specific features, and then composes new interaction samples in the feature space via stitching the decomposed features. The integration of decomposition and composition enables VCL to share object and verb features among different HOI samples and images, and to generate new interaction samples and new types of HOI, and thus largely alleviates the long-tail distribution problem and benefits low-shot or zero-shot HOI detection. Extensive experiments demonstrate that the proposed VCL can effectively improve the generalization of HOI detection on HICO-DET and V-COCO and outperforms the recent state-of-the-art methods on HICO-DET. Code is available at https://github.com/zhihou7/VCL.

show abstract

Section: Low-shot and Zero-shot Learningmentioning

confidence: 99%

“…Also, Yang et al [37] proposed to alleviate the predicate bias to objects for zero-shot visual relationship detection. Similar to previous approaches [3,26,29,36], we also equally treat the same verb from different HOIs. However, all those works [3,26,29,36] largely ignore the composition of verbs and objects.…”

Section: Low-shot and Zero-shot Learningmentioning

confidence: 99%

Visual Compositional Learning for Human-Object Interaction Detection

Huang

Peng

Qiao

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…One of the main problems of detecting visual relationships is the need for tremendous amounts of varied examples, as appearances and classes of both subject and target should vary for generalization of each interaction class. The release of large datasets [3,11,14,33] has allowed the developement of several visual relationship detectors in recent years [4,13,15,20,21,29,30,31,32] as well as HOI detectors [1,3,8,10,11,23,24,28].…”

Section: Related Workmentioning

confidence: 99%

“…Some techniques [1,28,29] incorporate linguistic knowledge to address the issue of having a long-tail distribution of human-object interaction classes. They exploit the contextual information present in the language priors learnt with a 'word2vec' network, to generalize interactions across functionally similar objects.…”

Section: Related Workmentioning

confidence: 99%

Classifying All Interacting Pairs in a Single Shot

Chafik¹,

Orcesi²,

Audigier³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we introduce a novel human interaction detection approach, based on CALIPSO (Classifying ALl Interacting Pairs in a Single shOt), a classifier of humanobject interactions. This new single-shot interaction classifier estimates interactions simultaneously for all humanobject pairs, regardless of their number and class. State-ofthe-art approaches adopt a multi-shot strategy based on a pairwise estimate of interactions for a set of human-object candidate pairs, which leads to a complexity depending, at least, on the number of interactions or, at most, on the number of candidate pairs. In contrast, the proposed method estimates the interactions on the whole image. Indeed, it simultaneously estimates all interactions between all human subjects and object targets by performing a single forward pass throughout the image. Consequently, it leads to a constant complexity and computation time independent of the number of subjects, objects or interactions in the image. In detail, interaction classification is achieved on a dense grid of anchors thanks to a joint multi-task network that learns three complementary tasks simultaneously: (i) prediction of the types of interaction, (ii) estimation of the presence of a target and (iii) learning of an embedding which maps interacting subject and target to a same representation, by using a metric learning strategy. In addition, we introduce an object-centric passive-voice verb estimation which significantly improves results. Evaluations on the two well-known Human-Object Interaction image datasets, V-COCO and HICO-DET, demonstrate the competitiveness of the proposed method (2nd place) compared to the state-ofthe-art while having constant computation time regardless of the number of objects and interactions in the image.This work proposes a new interaction detection approach, named CALIPSO (Classifying ALl Interacting Pairs in a Single shOt) which complexity is independent of the number of interactions. The proposed model simultaneously estimates all interactions between all objects with a

show abstract

“…Although modelling HOIs has been broadly studied in images [2,6,15,57], it has received less consideration in videos. Even deep learning methods have been developed for recognizing human actions in videos, most of them, including Covnet [41], recurrent neural networks (RNNs) [9,25] and 3D convolution models [5,49], only take individual frame-wise information as inputs (coarse-grained) without explicitly modeling (fine-grained) human-object relations across a video sequence.…”

Section: Introductionmentioning

confidence: 99%

Distillation of Human-Object Interaction Contexts for Action Recognition

Almushyti¹,

Li²

2021

Preprint

View full text Add to dashboard Cite

Modeling spatial-temporal relations is imperative for recognizing human actions, especially when a human is interacting with objects, while multiple objects appear around the human differently over time. Most existing action recognition models focus on learning overall visual cues of a scene but disregard informative fine-grained features, which can be captured by learning human-object relationships and interactions. In this paper, we learn human-object relationships by exploiting the interaction of their local and global contexts. We hence propose the Global-Local Interaction Distillation Network (GLIDN), learning human and object interactions through space and time via knowledge distillation for fine-grained scene understanding. GLIDN encodes humans and objects into graph nodes and learns local and global relations via graph attention network [52]. The local context graphs learn the relation between humans and objects at a frame level by capturing their cooccurrence at a specific time step. The global relation graph is constructed based on the video-level of human and object interactions, identifying their long-term relations throughout a video sequence. More importantly, we investigate how knowledge from these graphs can be distilled to their counterparts for improving human-object interaction (HOI) recognition. We evaluate our model by conducting comprehensive experiments on two datasets including Charades [40] and CAD-120 [24] datasets. We have achieved better results than the baselines and counterpart approaches.

show abstract

Detecting Human-Object Interactions via Functional Generalization

Cited by 4 publications

References 24 publications

Visual Compositional Learning for Human-Object Interaction Detection

Visual Compositional Learning for Human-Object Interaction Detection

Classifying All Interacting Pairs in a Single Shot

Distillation of Human-Object Interaction Contexts for Action Recognition

Contact Info

Product

Resources

About