Human object interaction detection: Design and survey

Antoun, Maya; Asmar, Daniel

doi:10.1016/j.imavis.2022.104617

Cited by 13 publications

(3 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To ensure training efficiency and stability, the batch size of the training samples was set to 10 and the proposed model was fully trained for 300 epochs. During the training process, previous research works are referred to [41][42][43] , and Stochastic Gradient Descent (SGD) 44 was chosen as the optimizer. SGD is a widely used optimization algorithm in machine learning that iteratively updates model parameters to minimize the loss function.…”

Section: Experimental Configuration and Parametersmentioning

confidence: 99%

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Zhang,

Yunos,

Haron

2024

Preprint

View full text Add to dashboard Cite

One of the challenges in training graph neural networks (GNNs) applied to human-object interaction (HOI) is the computational complexity associated with updating and aggregating the information of all connected nodes in dense graph data, which results in a long training time and poor convergence efficiency. In particular, the parallel multi-head graph attention network (PMGAT), a graph neural network model, has achieved promising results in HOI detection by capturing the interactive associations between keypoints through local feature modules and multi-head graph attention mechanisms. However, to address the challenge of computational complexity, this study proposes a graph sampling-based dynamic edge strategy called GraphSADES to improve the PMGAT. GraphSADES reduces computational complexity by dynamically sampling a subset of edges during the training process while maintaining the precision of the original model. Initially, an object-centered complete graph is constructed, node updates are performed to obtain the initial attention coefficients, and importance coefficients are computed. Subsequently, a dynamic edge sampling strategy is adopted to reduce the computational complexity by randomly selecting a subset of edges for updating and aggregating the information in each training step. Through experimental comparative analysis, GraphSADES-PMGAT maintains the precision of the PMGAT model, and the models are trained using ResNet-50 and ViT-B/16 as backbone networks. On the dataset, HICO-DET, Floating Point Operations (FLOPs) for computational complexity are decreased by 40.12% and 39.89%, and the training time is decreased by 14.20% and 12.02%, respectively, and the convergence efficiency is the earliest to converge after 180 epochs. On the V-COCO dataset, under the same backbone network condition as HICO-DET, FLOPs decreased by 39.81% and 39.56%, training time decreased by 10.26% and 16.91%, respectively, and the convergence efficiency was the earliest to converge after 165 epochs. Specifically, GraphSADES-PMGAT maintains comparable precision while reducing FLOPs, resulting in a shorter training time and improved convergence efficiency compared to the PMGAT model. This work opens up new possibilities for achieving efficient human-object interaction detection.

show abstract

Section: Experimental Configuration and Parametersmentioning

confidence: 99%

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Zhang,

Yunos,

Haron

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…HOI detection can facilitate the development of many tasks, such as anomaly detection [ 1 ], image understanding [ 2 ], and action recognition [ 3 ]. HOI detection methods [ 4 ] can be categorized into two-stage methods [ 5 , 6 , 7 ], traditional one-stage methods [ 8 , 9 , 10 ], and end-to-end one-stage methods [ 11 , 12 ]. Two-stage methods leverage an object detector to obtain confidence scores for humans and objects, subsequently filtering out low scores while retaining high-quality instances.…”

Section: Introductionmentioning

confidence: 99%

Pairwise CNN-Transformer Features for Human–Object Interaction Detection

Quan,

Lai,

Gao

et al. 2024

Entropy

View full text Add to dashboard Cite

Human–object interaction (HOI) detection aims to localize and recognize the relationship between humans and objects, which helps computers understand high-level semantics. In HOI detection, two-stage and one-stage methods have distinct advantages and disadvantages. The two-stage methods can obtain high-quality human–object pair features based on object detection but lack contextual information. The one-stage transformer-based methods can model good global features but cannot benefit from object detection. The ideal model should have the advantages of both methods. Therefore, we propose the Pairwise Convolutional neural network (CNN)-Transformer (PCT), a simple and effective two-stage method. The model both fully utilizes the object detector and has rich contextual information. Specifically, we obtain pairwise CNN features from the CNN backbone. These features are fused with pairwise transformer features to enhance the pairwise representations. The enhanced representations are superior to using CNN and transformer features individually. In addition, the global features of the transformer provide valuable contextual cues. We fairly compare the performance of pairwise CNN and pairwise transformer features in HOI detection. The experimental results show that the previously neglected CNN features still have a significant edge. Compared to state-of-the-art methods, our model achieves competitive results on the HICO-DET and V-COCO datasets.

show abstract

“…H UMAN-OBJECT interaction (HOI) detection is a process of detecting interaction between a human and an object in an image [1]. Precisely estimating human-object interactions can greatly improve various visual understanding tasks, such as image retrieval [2], visual question answering [3], and scene graph generation [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Zero-shot Object Detection Through Vision-Language Embedding Alignment

Xie

Zheng²

2022

2022 IEEE International Conference on Data Mining Workshops (ICDMW)

View full text Add to dashboard Cite

Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manual HOI annotations. In this paper, we propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to improve zero-shot HOI detection. Specifically, the verb feature learning module is designed based on visual semantics, by employing the verb extraction decoder to convert corresponding verb queries into interaction-specific category representations. We develop an effective additive self-attention mechanism to generate more comprehensive visual representations. Moreover, the innovative interaction representation decoder effectively extracts informative regions by integrating spatial and visual feature information through a cross-attention mechanism. To deal with zero-shot learning in low-data, we leverage a priori knowledge from the CLIP text encoder to initialize the linear classifier for enhanced interaction understanding. Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets demonstrate that our model outperforms the previous methods in various zero-shot and full-supervised settings.

show abstract

Human object interaction detection: Design and survey

Cited by 13 publications

References 36 publications

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Pairwise CNN-Transformer Features for Human–Object Interaction Detection

Zero-shot Object Detection Through Vision-Language Embedding Alignment

Contact Info

Product

Resources

About