Towards Hard-Positive Query Mining for DETR-Based Human-Object Interaction Detection

Zhong, Xubin; Ding, Changxing; Li, Zijian; Huang, Shaoli

doi:10.1007/978-3-031-19812-0_26

Cited by 18 publications

(7 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to the property of interaction queries employed, DETR-based one-stage methods can be classified into (i) approaches that adopt weight-fixed interaction queries during inference [6,42,54] and (ii) approaches that employ dynamic interaction queries [9,55,60,61]. The first class of methods are vulnerable to the weight-fixed and semantically unclear interaction queries.…”

Section: Related Workmentioning

confidence: 99%

“…To address this issue, Iftekhar et al [47] augmented the interaction queries with additional semantic and spatial cues; while Zhong et. al [54] improved the model's robustness by training DETR-based methods using hard-positive queries. The second class of methods adopt dynamic and semantically more clear interaction queries, which are usually obtained according to the human and object decoder embeddings.…”

Section: Related Workmentioning

confidence: 99%

“…One-Stage HOI-Trans [8] ResNet-101 52.9 AS-Net [42] ResNet-50 53.9 GGNet [39] Hourglass-104 54.7 HOTR [7] ResNet-50 55.2 QPIC [6] ResNet-50 58.8 CDN-S [9] ResNet-50 61.7 GEN-VLKTs [60] ResNet-50 62.4 DOQ [51] ResNet-50 63.5 HQM [54] ResNet-50…”

Section: Two-stagementioning

confidence: 99%

“…Human-Object Interaction (HOI) detection [9,17,54] aims to localize interactive human-object pairs in an image and identify the interaction categories between each human-object pair. The obtained results can be formulated as a set of HOI triplets <human, interaction, object>.…”

Section: Introductionmentioning

confidence: 99%

“…The downside of the two-stage methods lies in their low efficiency. In contrast, the recent popular one-stage methods extract the interaction representation efficiently [6,9,54]. As shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection

Zhong

Ding

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction; however, the interaction representations obtained using this method are entangled and lack interpretability. In contrast, traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner. In this paper, we improve the performance of one-stage methods by enabling them to extract disentangled interaction representations. First, we propose Shunted Cross-Attention (SCA) to extract human appearance, object appearance, and global context features using different cross-attention heads. This is achieved by imposing different masks on the cross-attention maps produced by the different heads. Second, we introduce the Interaction-aware Pose Estimation (IPE) task to learn interaction-relevant human pose features using a disentangled decoder. This is achieved with a novel attention module that accurately captures the human keypoints relevant to the current interaction category. Finally, our approach fuses the appearance feature and pose feature via element-wise addition to form the interaction representation. Experimental results show that our approach can be readily applied to existing one-stage HOI detectors. Moreover, we achieve state-of-the-art performance on two benchmarks: HICO-DET and V-COCO.

show abstract