A Multimodal Predictive Agent Model for Human Interaction Generation

Baruah, Murchana; Banerjee, Bonny

doi:10.1109/cvprw50498.2020.00519

Cited by 11 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While a number of models have been reported for intent prediction from body motions and/or eye gaze (see [ 17 , 18 ] for related reviews), few of them perform action classification and generation simultaneously. A large volume of work has been reported on generating actions using only one 3D skeleton (e.g., [ 15 , 19 , 20 , 21 ]) or on generating human motion in crowded scenes (e.g., [ 22 , 23 , 24 , 25 , 26 , 27 ]). Comparatively, much less has been reported on generating interaction of two persons using 3D skeletal data (e.g., [ 28 , 29 , 30 ]).…”

Section: Related Workmentioning

confidence: 99%

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Banerjee,

Baruah

2024

Sensors

Self Cite

View full text Add to dashboard Cite

The remarkable human ability to predict others’ intent during physical interactions develops at a very early age and is crucial for development. Intent prediction, defined as the simultaneous recognition and generation of human–human interactions, has many applications such as in assistive robotics, human–robot interaction, video and robotic surveillance, and autonomous driving. However, models for solving the problem are scarce. This paper proposes two attention-based agent models to predict the intent of interacting 3D skeletons by sampling them via a sequence of glimpses. The novelty of these agent models is that they are inherently multimodal, consisting of perceptual and proprioceptive pathways. The action (attention) is driven by the agent’s generation error, and not by reinforcement. At each sampling instant, the agent completes the partially observed skeletal motion and infers the interaction class. It learns where and what to sample by minimizing the generation and classification errors. Extensive evaluation of our models is carried out on benchmark datasets and in comparison to a state-of-the-art model for intent prediction, which reveals that classification and generation accuracies of one of the proposed models are comparable to those of the state of the art even though our model contains fewer trainable parameters. The insights gained from our model designs can inform the development of efficient agents, the future of artificial intelligence (AI).

show abstract

Section: Related Workmentioning

confidence: 99%

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Banerjee,

Baruah

2024

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some work [7,8] adopt RNN to synthesize humanhuman interaction given the partially observed interaction. [7] synthesized long-term interaction by alternatively generating the pose sequences of the two characters based on the generation history.…”

Section: Deep Generative Models In Motion Synthesismentioning

confidence: 99%

“…However, if one character approaches another character with a kick, then we may focus on the active leg and dodge at an appropriate timestamp. In synthesizing interactions, [8] attended to the informative joints to synthesize the reactive features which motivates our work to explore the synchronization of the two characters during the interaction.…”

Section: Attention Perceptionmentioning

confidence: 99%

“…As a result, the kinematic-based approaches [3,4] as well as combined enforcing kinematic and physical constraints [5,6] are used for preserving the context in editing close interaction in the literature. Existing work relevant to kinematics-based reactive motion synthesis mainly focus on generating interactions based on the interaction history [7,8], as well as synthesizing the response with non-parametric algorithms such as Markov Decision Process (MDP) [9,10,11] and motion blending [12,4]. However, it is a challenging task since the reactive motion is expected to respond properly and requires sufficient spatial and temporal synchronizations between the dynamics of the two characters, which can be potentially yet seldom explored by deep learning-based models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

Men¹,

Shum²,

Ho³

et al. 2021

Preprint

View full text Add to dashboard Cite

Creating realistic characters that can react to the users' or another character's movement can benefit computer graphics, games and virtual reality hugely. However, synthesizing such reactive motions in human-human interactions is a challenging task due to the many different ways two humans can interact. While there are a number of successful researches in adapting the generative adversarial network (GAN) in synthesizing single human actions, there are very few on modelling human-human interactions. In this paper, we propose a semi-supervised GAN system that synthesizes the reactive motion of a character given the active motion from another character. Our key insights are two-fold. First, to effectively encode the complicated spatial-temporal information of a human motion, we empower the generator with a part-based long short-term memory (LSTM) module, such that the temporal movement of different limbs can be effectively modelled. We further include an attention module such that the temporal significance of the interaction can be learned, which enhances the temporal alignment of the active-reactive motion pair. Second, as the reactive motion of different types of interactions can be significantly different, we introduce a discriminator that not only tells if the generated movement is realistic or not, but also tells the class label of the interaction. This allows the use of such labels in supervising the training of the generator. We experiment with the SBU and the HHOI datasets. The high quality of the synthetic motion demonstrates the effective design of our generator, and the discriminability of the synthesis also demonstrates the strength of our discriminator.

show abstract

“…1 An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators [2]. Such agents, implemented in software, have been reported in our prior work [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] as well as in others'.…”

Section: Introductionmentioning

confidence: 99%

Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

Banerjee¹,

Kapourchali²,

Baruah³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Round-the-clock monitoring of human behavior and emotions is required in many healthcare applications which is very expensive but can be automated using machine learning (ML) and sensor technologies. Unfortunately, the lack of infrastructure for collection and sharing of such data is a bottleneck for ML research applied to healthcare. Our goal is to circumvent this bottleneck by simulating a human body in virtual environment. This will allow generation of potentially infinite amounts of shareable data from an individual as a function of his actions, interactions and emotions in a care facility or at home, with no risk of confidentiality breach or privacy invasion. In this paper, we develop for the first time a system consisting of computational models for synchronously synthesizing skeletal motion, electrocardiogram, blood pressure, respiration, and skin conductance signals as a function of an open-ended set of actions and emotions. Our experimental evaluations, involving user studies, benchmark datasets and comparison to findings in the literature, show that our models can generate skeletal motion and physiological signals with high fidelity. The proposed framework is modular and allows the flexibility to experiment with different models. In addition to facilitating ML research for round-the-clock monitoring at a reduced cost, the proposed framework will allow reusability of code and data, and may be used as a training tool for ML practitioners and healthcare professionals.

show abstract

A Multimodal Predictive Agent Model for Human Interaction Generation

Cited by 11 publications

References 22 publications

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

Contact Info

Product

Resources

About