IPGN: Interactiveness Proposal Graph Network for Human-Object Interaction Detection

Wang, Haoran; Jiao, Licheng; Liu, Fang; Liu, Xu; Ji, Deyi; Gan, Weihao

doi:10.1109/tip.2021.3096333

Cited by 24 publications

(4 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where [•, •] denotes the concatenation operation. In the connection graph, which contains human-object connections with and without interactions, if Equation ( 7) is used directly when integrating the features of neighboring nodes, the number of connections with invalid interactions is greater than the number of connections with valid interactions [47], i.e., the number of noises is too large; thus, this algorithm introduces an attention mechanism to reduce the interference of invalid interactions:…”

Section: Graph Modelmentioning

confidence: 99%

Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection

2023

View full text Add to dashboard Cite

Human–object interaction (HOI) detection is important for promoting the development of many fields such as human–computer interactions, service robotics, and video security surveillance. A high percentage of human–object pairs with invalid interactions are discovered in the object detection phase of conventional human–object interaction detection algorithms, resulting in inaccurate interaction detection. To recognize invalid human–object interaction pairs, this paper proposes a model structure, the interactivity recognition graph neural network (IR-GNN) model, which can directly infer the probability of human–object interactions from a graph model architecture. The model consists of three modules: The first one is the human posture feature module, which uses key points of the human body to construct relative spatial pose features and further facilitates the discrimination of human–object interactivity through human pose information. Second, a human–object interactivity graph module is proposed. The spatial relationship of human–object distance is used as the initialization weight of edges, and the graph is updated by combining the message passing of attention mechanism so that edges with interacting node pairs obtain higher weights. Thirdly, the classification module is proposed; by finally using a fully connected neural network, the interactivity of human–object pairs is binarily classified. These three modules work in collaboration to enable the effective inference of interactive possibilities. On the datasets HICO-DET and V-COCO, comparative and ablation experiments are carried out. It has been proved that our technology can improve the detection of human–object interactions.

show abstract

Section: Graph Modelmentioning

confidence: 99%

Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection

2023

View full text Add to dashboard Cite

show abstract

“…Binary Image Representation is widely utilized to model the spatial relation [4], [10], [27]- [29]. It characterizes the overlap or intersection relations among objects.…”

Section: Binary Image Representationmentioning

confidence: 99%

Spatial-Semantic Transformer for Spatial Relation Recognition

Peng,

Wang,

2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Spatial relation recognition, which aims to predict a spatial relation predicate, has attracted increasing attention in the computer vision study. During tackling this problem, modeling spatial relation of the subjects and objects is of great importance. We find that only using spatial features leads to poor results in predicting the spatial relation. To overcome these challenges, we propose an effective spatial attention module to enhance spatial features using semantic features. After identifying the importance of spatial attention mechanism, we propose a spatial transformer module with encoder layers to recognize unseen spatial relation based on spatial attention mechanism. Extensive experiments on the benchmark dataset (SpatialSense) show that, by using refined spatial feature, our spatial transformer model and spatial attention model achieve state-of-the-art performance on overall accuracy.

show abstract

“…This task is crucial for scene understanding and has a wide range of applications in the fields of action recognition, robotics, and human-computer interaction [4][5][6] . Recently, graph neural networks (GNNs) have emerged as a powerful tool for HOI detection because of their ability to efficiently capture complex relationships between entities in graph-structured data [7][8][9][10][11][12][13][14][15][16] . The GNN provides a new perspective for HOI detection by capturing entity relationships in graphstructured data.…”

Section: Introductionmentioning

confidence: 99%

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Zhang,

Yunos,

Haron

2024

Preprint

View full text Add to dashboard Cite

One of the challenges in training graph neural networks (GNNs) applied to human-object interaction (HOI) is the computational complexity associated with updating and aggregating the information of all connected nodes in dense graph data, which results in a long training time and poor convergence efficiency. In particular, the parallel multi-head graph attention network (PMGAT), a graph neural network model, has achieved promising results in HOI detection by capturing the interactive associations between keypoints through local feature modules and multi-head graph attention mechanisms. However, to address the challenge of computational complexity, this study proposes a graph sampling-based dynamic edge strategy called GraphSADES to improve the PMGAT. GraphSADES reduces computational complexity by dynamically sampling a subset of edges during the training process while maintaining the precision of the original model. Initially, an object-centered complete graph is constructed, node updates are performed to obtain the initial attention coefficients, and importance coefficients are computed. Subsequently, a dynamic edge sampling strategy is adopted to reduce the computational complexity by randomly selecting a subset of edges for updating and aggregating the information in each training step. Through experimental comparative analysis, GraphSADES-PMGAT maintains the precision of the PMGAT model, and the models are trained using ResNet-50 and ViT-B/16 as backbone networks. On the dataset, HICO-DET, Floating Point Operations (FLOPs) for computational complexity are decreased by 40.12% and 39.89%, and the training time is decreased by 14.20% and 12.02%, respectively, and the convergence efficiency is the earliest to converge after 180 epochs. On the V-COCO dataset, under the same backbone network condition as HICO-DET, FLOPs decreased by 39.81% and 39.56%, training time decreased by 10.26% and 16.91%, respectively, and the convergence efficiency was the earliest to converge after 165 epochs. Specifically, GraphSADES-PMGAT maintains comparable precision while reducing FLOPs, resulting in a shorter training time and improved convergence efficiency compared to the PMGAT model. This work opens up new possibilities for achieving efficient human-object interaction detection.

show abstract

IPGN: Interactiveness Proposal Graph Network for Human-Object Interaction Detection

Cited by 24 publications

References 54 publications

Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection

Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection

Spatial-Semantic Transformer for Spatial Relation Recognition

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

Contact Info

Product

Resources

About