Reasoning About Human-Object Interactions Through Dual Attention Networks

Xiao, Tete; Fan, Quanfu; Gutfreund, Dan; Monfort, Mathew; Oliva, Aude; Zhou, Bolei

doi:10.1109/iccv.2019.00402

Cited by 37 publications

(19 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Attention mechanisms have provided and will provide a paradigm shift in machine learning. Specifically, this change is from traditional large-scale vector transformations to more conscious processes (i.e., that focus only on a set of elements), e.g., decomposing a problem into a sequence of attention based reasoning tasks [13,[30][31][32][33][34].…”

Section: Differentiable Attentionmentioning

confidence: 99%

Attention Mechanisms and Their Applications to Complex Systems

Hernandez

Amigó

2021

Entropy

View full text Add to dashboard Cite

Deep learning models and graphics processing units have completely transformed the field of machine learning. Recurrent neural networks and long short-term memories have been successfully used to model and predict complex systems. However, these classic models do not perform sequential reasoning, a process that guides a task based on perception and memory. In recent years, attention mechanisms have emerged as a promising solution to these problems. In this review, we describe the key aspects of attention mechanisms and some relevant attention techniques and point out why they are a remarkable advance in machine learning. Then, we illustrate some important applications of these techniques in the modeling of complex systems.

show abstract

Section: Differentiable Attentionmentioning

confidence: 99%

Attention Mechanisms and Their Applications to Complex Systems

Hernandez

Amigó

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…interaction observation to improve object recognition [40]- [42], or used to learn semantics and boost object localization for improved scene understanding [43], [44]. In action understanding, object affordances have been utilized for action anticipation [45]- [49], hand grasp generation [50], [51], and used as context information to improve action recognition [52], [53].…”

Section: A Affordance As Auxiliary Informationmentioning

confidence: 99%

Joint Object Affordance Reasoning and Segmentation in RGB-D Videos

2021

View full text Add to dashboard Cite

The authors would like to thank the European Commission for financial support through H2020 project HR-Recycler under contract 820742 and Nvidia Corporation for a Titan X GPU donation ABSTRACT Understanding human-object interaction is a fundamental challenge in computer vision and robotics. Crucial to it is the ability to infer "object affordances" from visual data, namely the types of interaction supported by an object of interest and the object parts involved. Such inference can be approached as an "affordance reasoning" task, where object affordances are recognized and localized as image heatmaps, and as an "affordance segmentation" task, where affordance labels are obtained at a more detailed, image pixel level. To tackle the two tasks, existing methods typically: (i) treat them independently;(ii) adopt static image-based models, ignoring the temporal aspect of human-object interaction; and / or (iii) require additional strong supervision concerning object class and location. In this paper, we focus on both tasks, while addressing all three aforementioned shortcomings. For this purpose, we propose a deep-learning based dual encoder-decoder model for joint affordance reasoning and segmentation, which learns from our recently introduced SOR3D-AFF corpus of RGB-D human-object interaction videos, without relying on object localization and classification. The basic components of the model comprise: (i) two parallel encoders that capture spatio-temporal interaction information; (ii) a reasoning decoder that predicts affordance heatmaps, assisted by an affordance classifier and an attention mechanism; and (iii) a segmentation decoder that exploits the predicted heatmap to yield pixel-level affordance segmentation. All modules are jointly trained, while the system can operate on both static images and videos. The approach is evaluated on four datasets, surpassing the current state-of-the-art in both affordance reasoning and segmentation.

show abstract

“…Rahmani et al [36] proposed the robust nonlinear knowledge transfer model (R-NKTM) for human action recognition. Xiao et al [37] proposed a dual attention network model which reasons about human-object interactions. is network weighs the important features for objects and actions, respectively.…”

Section: Deep Interactionmentioning

confidence: 99%

Multiperson Interactive Activity Recognition Based on Interaction Relation Model

Chi

et al. 2021

Journal of Mathematics

View full text Add to dashboard Cite

Multiperson activity recognition is a pivotal branch as well as a challenging topic of human action recognition research. This paper adopts a hybrid learning model to the spatio-temporal relationship and occlusion relationship among multiple people. Initially, this paper builds up an active multiperson interaction relationship estimation framework model to capture interpersonal spatio-temporal relation. This model incorporates the interaction relationship estimation framework with the multiperson relationship network. On this ground, it automatically learns from the human-computer interaction dataset in an end-to-end manner and performs reasoning with standard matrix operations. Secondly, this paper proposed an adaptive occlusion state behavior recognition method derived from the semantic knowledge model to ravel out the concern of occlusion and self-occlusion in human action recognition. Then, Petri Nets are used to recognize multiperson interactive actions. This model has been through extensive experiments on the TV interaction dataset, Vlog dataset, AVA dataset, and MLB-YouTube dataset, experimental results have proved that the recognition performance of this model is superior than the other available models. This paper prospects and summarizes the estimation framework of the interaction relationship and occlusion semantic-knowledge relationship. Experimental results suggest that the proposed method in the paper could capture the discriminative relation information for multiperson interactive activity recognition, which further validates the efficiency of the hybrid learning model.

show abstract

Reasoning About Human-Object Interactions Through Dual Attention Networks

Abstract: Objects are entities we act upon, where the functionality of an object is determined by how we interact with it.

Cited by 37 publications

References 45 publications

Attention Mechanisms and Their Applications to Complex Systems

Attention Mechanisms and Their Applications to Complex Systems

Joint Object Affordance Reasoning and Segmentation in RGB-D Videos

Multiperson Interactive Activity Recognition Based on Interaction Relation Model

Contact Info

Product

Resources

About