GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Liao, Yue; Zhang, Aixi; Lu, Miao; Wang, Yongliang; Li, Xiaobo; Liu, Si

doi:10.1109/cvpr52688.2022.01949

Cited by 53 publications

(33 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, transformer-based HOI detection has gained significant attention and demonstrated promising results, primarily due to QPIC's innovative use of transformer-based techniques in HOI detection. Subsequently, several works have followed in the footsteps of QPIC, such as [4], [20], [34]. However, it is more comprehensive than one-stage approaches adopting transformer-based architectures, as demonstrated by using transformer-based techniques in the two-stage HOI detection method UPT [37].…”

Section: A Human-object Interaction Detectionmentioning

confidence: 99%

“…These methods can be broadly classified into one-stage and twostage approaches. In the case of one-stage approaches [5], [32], [41], [20], [24], the task is accomplished in a single step without the need for detecting the instance first. They can simultaneously detect all pairs of instances on an image and the interaction between these instance pairs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

Zhu¹,

Lan²,

Velasquez³

et al. 2023

Preprint

View full text Add to dashboard Cite

Detecting human-object interactions (HOIs) is a challenging problem in computer vision. Existing techniques for HOI detection heavily rely on appearance-based features, which may not capture other essential characteristics for accurate detection. Furthermore, the use of transformer-based models for sentiment representation of human-object pairs can be computationally expensive. To address these challenges, we propose a novel graph-based approach, SKGHOI (Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection), that effectively captures the sentiment representation of HOIs by integrating both spatial and semantic knowledge. In a graph, SKGHOI takes the components of interaction as nodes, and the spatial relationships between them as edges. Our approach employs a spatial encoder and a semantic encoder to extract spatial and semantic information, respectively, and then combines these encodings to create a knowledge graph that captures the sentiment representation of HOIs. Compared to existing techniques, SKGHOI is computationally efficient and allows for the incorporation of prior knowledge, making it practical for use in real-world applications. We demonstrate the effectiveness of our proposed method on the widely-used HICO-DET datasets, where it outperforms existing state-of-the-art graph-based methods by a significant margin. Our results indicate that the SKGHOI approach has the potential to significantly improve the accuracy and efficiency of HOI detection, and we anticipate that it will be of great interest to researchers and practitioners working on this challenging task.

show abstract

Section: A Human-object Interaction Detectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

Zhu¹,

Lan²,

Velasquez³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…UPT [19] applied a unary-pairwise transformer to represent each target's instance details as unary and pairwise representations. In comparison to two-stage methods, one-stage solutions [9], [20]- [26] captured context information during the early stage of feature extraction, leading to improved HOI detection performance. The success of DETR [27] has inspired many researchers in studying HOI detection QPIC [28] applied additional detection heads and relied on a bipartite graph matching algorithm to locate HOI instances and identify interactions.…”

Section: A Human-object Interaction Detectionmentioning

confidence: 99%

“…RLIP [23] proposed a transferable HOI detector via natural language supervision. Building upon GEN-VLKT [20], HOICLIP [29] mapped image and text encodings to a joint visual-semantic space, to capture their correlations and effectively transfer knowledge from CLIP. Our work seeks to explore a more effective framework to make full use of CLIP for improving zero-shot HOI detection performance.…”

Section: Zero-shot Hoimentioning

confidence: 99%

“…Given an input image I, we initially extract feature maps via the backbone DETR [27]. The feature maps are subsequently input into the HO-pair Encoder to generate global visual feature V g , similar to GEN-VLKT [20]. Human query Q h and object query Q o are inputted into the instance interactor to compute the mean for both types of queries in the corresponding decoder layer.…”

Section: A Overall Architecturementioning

confidence: 99%

See 1 more Smart Citation

Zero-shot Object Detection Through Vision-Language Embedding Alignment

Xie

Zheng²

2022

2022 IEEE International Conference on Data Mining Workshops (ICDMW)

View full text Add to dashboard Cite

Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manual HOI annotations. In this paper, we propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to improve zero-shot HOI detection. Specifically, the verb feature learning module is designed based on visual semantics, by employing the verb extraction decoder to convert corresponding verb queries into interaction-specific category representations. We develop an effective additive self-attention mechanism to generate more comprehensive visual representations. Moreover, the innovative interaction representation decoder effectively extracts informative regions by integrating spatial and visual feature information through a cross-attention mechanism. To deal with zero-shot learning in low-data, we leverage a priori knowledge from the CLIP text encoder to initialize the linear classifier for enhanced interaction understanding. Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets demonstrate that our model outperforms the previous methods in various zero-shot and full-supervised settings.

show abstract

Spatial-Temporal Transformer Network for Continuous Action Recognition in Industrial Assembly

Huang,

Liu,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Cited by 53 publications

References 24 publications

SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

Zero-shot Object Detection Through Vision-Language Embedding Alignment

Spatial-Temporal Transformer Network for Continuous Action Recognition in Industrial Assembly

Contact Info

Product

Resources

About