Generative Modeling of Multimodal Multi-Human Behavior

Abstract

This work presents a methodology for modeling and predicting human behavior in settings with N humans interacting in highly multimodal scenarios (i.e. where there are many possible highly-distinct futures). A motivating example includes robots interacting with humans in crowded environments, such as self-driving cars operating alongside humandriven vehicles or human-robot collaborative bin packing in a warehouse. Our approach to model human behavior in such uncertain environments is to model humans in the scene as nodes in a graphical model, with edges encoding relationships between them. For each human, we learn a multimodal probability distribution over future actions from a dataset of multi-human interactions. Learning such distributions is made possible by recent advances in the theory of conditional variational autoencoders and deep learning approximations of probabilistic graphical models. Specifically, we learn action distributions conditioned on interaction history, neighboring human behavior, and candidate future agent behavior in order to take into account response dynamics. We demonstrate the performance of such a modeling approach in modeling basketball player trajectories, a highly multimodal, multi-human scenario which serves as a proxy for many robotic applications.

PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings

“…Third, we model interactions between more than two vehicles jointly. While [15] assumes conditional independencies for computational reasons, we do not, as they impose minimal overhead.…”

“…withL parameterized by (q, f, p), and f 's dependence on φ dropped for notational brevity. The 1:K z h is redrawn before each gradient ascent step on (15). This procedure is illustrated in Alg.…”

“…In our implementation, we use K = 12, track the z r 1:T that achieved the bestL score, terminate the ascent if the best z r 1:T has not improved in 10 steps, and return the corresponding best z r 1:T . To initialize z r 1:T more robustly, we sample a full 1:K z multiple times (15), and use the z r corresponding to the bestL. We can also run the planning over multiple initial samples of z r 1:T .…”

The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs

“…1). Nodes represent agents and we form edges based on agents' spatial proximity, as in prior work [1,24].…”

“…We combine the states of all neighboring nodes of a specific edge type by summing them and feeding the result into the appropriate edge encoder, obtaining an edge influence representation. We choose to combine representations in this manner rather than via concatenation in order to handle a variable number of neighboring nodes with a fixed architecture while preserving count information [6,24,25]. These representations are then passed through scalar multiplications that modulate the outputs of EEs depending on the age of an edge.…”