Improving Human-Object Interaction Detection via Phrase Learning and Label Composition

Li, Zhimin; Zhang, Cheng; Yu, Zhiwen; Li, Boxun; Su, Zhong

doi:10.48550/arxiv.2112.07383

Cited by 2 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing HOI detection methods can be roughly categorized into crop-based and crop-free fashions [18,19,20,21].…”

Section: Related Workmentioning

confidence: 99%

Effective Actor-centric Human-object Interaction Detection

Xu,

Li,

Zhang

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

While Human-Object Interaction(HOI) Detection has achieved tremendous advances in recent, it still remains challenging due to complex interactions with multiple humans and objects occurring in images, which would inevitably lead to ambiguities. Most existing methods either generate all human-object pair candidates and infer their relationships by cropped local features successively in a two-stage manner, or directly predict interaction points in a one-stage procedure. However, the lack of spatial configurations or reasoning steps of two-or one-stage methods respectively limits their performance in such complex scenes. To avoid this ambiguity, we propose a novel actor-centric framework. The main ideas are that when inferring interactions: 1) the non-local features of the entire image guided by actor position are obtained to model the relationship between the actor and context, and then 2) we use an object branch to generate pixel-wise interaction area prediction, where the interaction area denotes the object central area. Moreover, we also use an actor branch to get interaction prediction of the actor and propose a novel composition strategy based on center-point indexing to generate the final HOI prediction. Thanks to the usage of the non-local features and the partly-coupled property of the human-objects composition strategy, our proposed framework can detect HOI more accurately especially for complex images. Extensive experimental results show that our method achieves the state-of-the-art on the challenging V-COCO and HICO-DET benchmarks and is more robust especially in multiple persons and/or objects scenes.

show abstract

“…Existing HOI detection methods can be roughly categorized into crop-based and crop-free fashions [18,19,20,21].…”

Section: Related Workmentioning

confidence: 99%

Effective Actor-centric Human-object Interaction Detection

Xu,

Li,

Zhang

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Conventional HOI detectors are mainly divided into two folds, bottom-up and top-down. The bottomup pipelines [3,6,7,10,18,19,21,22,24,35,38,45] first detect all humans and objects and then associate the human-object pairs and infer their HOI types through an additional classifier. These methods are usually organized as a two-stage paradigm and worked on improving the second stage.…”

Section: Related Workmentioning

confidence: 99%

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Liao¹,

Zhang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The task of Human-Object Interaction (HOI) detection could be divided into two core problems, i.e., human-object association and interaction understanding. In this paper, we reveal and address the disadvantages of the conventional query-driven HOI detectors from the two aspects. For the association, previous two-branch methods suffer from complex and costly post-matching, while single-branch methods ignore the features distinction in different tasks. We propose Guided-Embedding Network (GEN) to attain a twobranch pipeline without post-matching. In GEN, we design an instance decoder to detect humans and objects with two independent query sets and a position Guided Embedding (p-GE) to mark the human and object in the same position as a pair. Besides, we design an interaction decoder to classify interactions, where the interaction queries are made of instance Guided Embeddings (i-GE) generated from the outputs of each instance decoder layer. For the interaction understanding, previous methods suffer from long-tailed distribution and zero-shot discovery. This paper proposes Visual-Linguistic Knowledge Transfer (VLKT) training strategy to enhance interaction understanding by transferring knowledge from a visual-linguistic pre-trained model CLIP. In specific, we extract text embeddings for all labels with CLIP to initialize the classifier and adopt a mimic loss to minimize the visual feature distance between GEN and CLIP. As a result, GEN-VLKT outperforms the state of the art by large margins on multiple datasets, e.g., +5.05 mAP on HICO-Det. The source codes are available at https://github.com/YueLiao/gen-vlkt.

show abstract

Improving Human-Object Interaction Detection via Phrase Learning and Label Composition

Cited by 2 publications

References 20 publications

Effective Actor-centric Human-object Interaction Detection

Effective Actor-centric Human-object Interaction Detection

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Contact Info

Product

Resources

About