End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation

Abstract

Most existing Human-Object Interaction (HOI) Detection methods rely heavily on full annotations with predefined HOI categories, which is limited in diversity and costly to scale further. We aim at advancing zero-shot HOI detection to detect both seen and unseen HOIs simultaneously. The fundamental challenges are to discover potential human-object pairs and identify novel HOI categories. To overcome the above challenges, we propose a novel End-to-end zero-shot HOI Detection (EoID) framework via vision-language knowledge distillation. We first design an Interactive Score module combined with a Two-stage Bipartite Matching algorithm to achieve interaction distinguishment for human-object pairs in an action-agnostic manner. Then we transfer the distribution of action probability from the pretrained vision-language teacher as well as the seen ground truth to the HOI model to attain zero-shot HOI classification. Extensive experiments on HICO-Det dataset demonstrate that our model discovers potential interactive pairs and enables the recognition of unseen HOIs. Finally, our method outperforms the previous SOTA under various zero-shot settings. Moreover, our method is generalizable to large-scale object detection data to further scale up the action sets. The source code is available at: https://github.com/mrwu-mac/EoID.

Zero-shot Object Detection Through Vision-Language Embedding Alignment

“…UPT [19] applied a unary-pairwise transformer to represent each target's instance details as unary and pairwise representations. In comparison to two-stage methods, one-stage solutions [9], [20]- [26] captured context information during the early stage of feature extraction, leading to improved HOI detection performance. The success of DETR [27] has inspired many researchers in studying HOI detection QPIC [28] applied additional detection heads and relied on a bipartite graph matching algorithm to locate HOI instances and identify interactions.…”

“…The success of DETR [27] has inspired many researchers in studying HOI detection QPIC [28] applied additional detection heads and relied on a bipartite graph matching algorithm to locate HOI instances and identify interactions. EOID [9] developed a teacher-student model and designed a two-stage Hungarian matching algorithm. RR-Net [26] introduced a relation-aware frame to build progressive structure for interaction inference, which imitates the human visual mechanism of recognizing HOI by comprehending visual instances and interactions coherently.…”

“…The most recent zero-shot HOI detectors leverage the comprehensive visual and linguistic knowledge of CLIP to detect novel HOIs. For example, the knowledge of pre-trained visual language model is transferred to EOID [9] and DOQ [10] via knowledge distillation to achieve zero-shot HOI detection. Previous approaches utilize cross-modal knowledge in a toorestrictive manner, failing to fully capitalize on the potential of cross-modal information and Language Models in the field of Human-Object Interaction (HOI) detection.…”

Turning a CLIP Model into a Scene Text Detector

“…Recently, developments in leveraging pretrained vision and language knowledge, particularly through the large-scale Contrastive Language-Image Pretraining (CLIP) model [1], have shown promising results in a wide range of downstream tasks. These include but are not limited to image classification [2], object detection [3], and semantic segmentation [4], [5]. In the realm of text spotting, where scene text often provides rich visual and character information, the potential of the CLIP model is particularly evident.…”

Key Technology of Automation Control Based on Artificial Intelligence Technology

“…The current prevailing trend of transferring multimodal knowledge in vision-language pretraining models [44,23,30,62] to a diverse range of downstream tasks has been proven effective and achieved remarkable success [85,15,25,58]. A natural question arises: How can few-shot action recognition take advantage of the foundation model to mine the powerful multimodal knowledge?…”