SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

Abstract

Detecting human-object interactions (HOIs) is a challenging problem in computer vision. Existing techniques for HOI detection heavily rely on appearance-based features, which may not capture other essential characteristics for accurate detection. Furthermore, the use of transformer-based models for sentiment representation of human-object pairs can be computationally expensive. To address these challenges, we propose a novel graph-based approach, SKGHOI (Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection), that effectively captures the sentiment representation of HOIs by integrating both spatial and semantic knowledge. In a graph, SKGHOI takes the components of interaction as nodes, and the spatial relationships between them as edges. Our approach employs a spatial encoder and a semantic encoder to extract spatial and semantic information, respectively, and then combines these encodings to create a knowledge graph that captures the sentiment representation of HOIs. Compared to existing techniques, SKGHOI is computationally efficient and allows for the incorporation of prior knowledge, making it practical for use in real-world applications. We demonstrate the effectiveness of our proposed method on the widely-used HICO-DET datasets, where it outperforms existing state-of-the-art graph-based methods by a significant margin. Our results indicate that the SKGHOI approach has the potential to significantly improve the accuracy and efficiency of HOI detection, and we anticipate that it will be of great interest to researchers and practitioners working on this challenging task.

Parallel Multi-Head Graph Attention Network (PMGAT) Model for Human-Object Interaction Detection

“…While the Transformer has achieved good results in HOI detection, it requires significant computational resources, posing challenges for subsequent inference tasks. However, alternatives to the transformer, such as graph-based techniques, have been suggested to tackle its limitations and can be considered a viable approach [19], [40], [41], [42].…”

“…Through a multi-branch fusion mechanism, SCG utilizes the spatial arrangement of person-object pairs to adjust appearance features, improve edge computations, and thereby enhance the quality of HOI detection. TMHOI: Zhu et al [41] proposed TMHOI, a method that utilizes a knowledge graph embedding model as a translation model. The purpose is to incorporate relationship features into node embeddings through embedding and integration.…”

“…This is defined in Equation ( 22). The Equation is based on a few studies by [41] and [34]. 23), our training objective function uses the common loss function used in [18].…”

“…It should be noted that PMGAT also achieves state-of-the-art performance and outperforms existing state-of-the-art approaches when using the same backbone network support as ViPLO. ResNet-50-FPN 51.0 -SCG [40] ResNet-50-FPN 54.2 60.9 TMHOI [41] ResNet-50-FPN --ViPLO [42] ViT-B/16 For the four different backbone networks, the performance of the proposed PMGAT was tested on the HICO-DET dataset, and the results are presented in Table 3. The test results indicate that as the depth of the ResNet backbone increases from 18 to 34, and then to 50 and 101, there is a significant overall performance improvement for both situations.…”

“…Transferable Interactiveness Knowledge (TIN) [21], Dual Relation Graph (DRG) [19], Spatially Conditioned Graphs (SCG) [40], Translational Model for Human-Object Interaction Detection (TMHOI) [41], and Vision Transformer based Pose-Conditioned Self-Loop Graph (ViPLO) [42], have been applied to HOI detection on the HICO-DET dataset, and the recorded recall rates have been documented.…”