Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection

Abstract

Human–object interaction (HOI) detection is important for promoting the development of many fields such as human–computer interactions, service robotics, and video security surveillance. A high percentage of human–object pairs with invalid interactions are discovered in the object detection phase of conventional human–object interaction detection algorithms, resulting in inaccurate interaction detection. To recognize invalid human–object interaction pairs, this paper proposes a model structure, the interactivity recognition graph neural network (IR-GNN) model, which can directly infer the probability of human–object interactions from a graph model architecture. The model consists of three modules: The first one is the human posture feature module, which uses key points of the human body to construct relative spatial pose features and further facilitates the discrimination of human–object interactivity through human pose information. Second, a human–object interactivity graph module is proposed. The spatial relationship of human–object distance is used as the initialization weight of edges, and the graph is updated by combining the message passing of attention mechanism so that edges with interacting node pairs obtain higher weights. Thirdly, the classification module is proposed; by finally using a fully connected neural network, the interactivity of human–object pairs is binarily classified. These three modules work in collaboration to enable the effective inference of interactive possibilities. On the datasets HICO-DET and V-COCO, comparative and ablation experiments are carried out. It has been proved that our technology can improve the detection of human–object interactions.

Improved PMGAT for Human-Object Interaction Detection through Graph Sampling-based Dynamic Edge Strategy (GraphSADES)

“…GNN has unique advantages in modelling the interaction between the human body and objects. For example, STIGPN 12 and TMHOI 23 have successfully captured spatial and temporal variations in interaction behavior by constructing dynamic or relational graphs, enabling accurate identification of HOI behavior. Fusion of multi-modal information.…”

“…Fusion of multi-modal information. Studies by Zhang et al 24 and Fanuel et al 25 have explored fusing multi-modal information, such as vision and pose, into GNNs. The model can understand the interaction behavior more comprehensively by introducing additional information sources, thus improving detection precision.…”

From Interaction Data to Personalized Learning: Mining User-Object Interactions in Intelligent Environments

“…Different research projects have studied human-object interaction detection. One example is the improvement of the accuracy while detecting interactions between persons and objects using computer vision and a graph neural network [12], or a graph model-based algorithm [13]; an application for smart glasses that assists workers in an industrial site recognizing human-object interactions [14]; as a contribution in the visual understanding field [15][16], or to help to solve the problem of missing human behavior objects [17]. It can be observed that behavior detection is related to the technologies available in the intelligent environment configuration, the sensors installed in the background, or can be related to the data sources, such as the case of images, videos, interactions with software systems [18], cameras for collecting images, microphones to record sounds within the classroom [19], and the dialogue between student and teacher and wearable devices [20] through which it is possible to identify tasks and student behavior in a classroom or digital learning environments as intelligent tutors.…”

Parallel Multi-Head Graph Attention Network (PMGAT) Model for Human-Object Interaction Detection

“…In the HOIPM, we employed IR-GNN [48] to perform this task. The IR-GNN model employs a graph-based structure, enabling the effective estimation of interaction probabilities between humans and a multitude of objects.…”