Learning Temporal Transformations from Time-Lapse Videos

111

193

We present a new approach to modeling visual attributes. Prior work casts attributes in a similar role as objects, learning a latent representation where properties (e.g., sliced) are recognized by classifiers much in the way objects (e.g., apple) are. However, this common approach fails to separate the attributes observed during training from the objects with which they are composed, making it ineffectual when encountering new attribute-object compositions. Instead, we propose to model attributes as operators. Our approach learns a semantic embedding that explicitly factors out attributes from their accompanying objects, and also benefits from novel regularizers expressing attribute operators' effects (e.g., blunt should undo the effects of sharp). Not only does our approach align conceptually with the linguistic role of attributes as modifiers, but it also generalizes to recognize unseen compositions of objects and attributes. We validate our approach on two challenging datasets and demonstrate significant improvements over the state of the art. In addition, we show that not only can our model recognize unseen compositions robustly in an open-world setting, it can also generalize to compositions where objects themselves were unseen during training.

Section: Related Workmentioning

confidence: 99%

Attributes as Operators: Factorizing Unseen Attribute-Object Compositions

Nagarajan

Grauman

2018

111

193

“…Mikolov et al [38] showcased the composition additive property of word vectors learned in an unsupervised way from language data; Kulkarni et al [39], Reed et al [40] suggested that additive transformation can be achieved via reconstruction or prediction task by learning from parallel paired image data. In the video domain, Wang et al [41] studied a transformation-aware representation for semantic human action classification; Zhou et al [42] investigated time-lapse video generation given additional class labels.…”

Section: Related Workmentioning

confidence: 99%

MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics

Yan

Rastogi

Villegas

et al. 2018

111

Long-term human motion can be represented as a series of motion modes-motion sequences that capture short-term temporal dynamics-with transitions between them. We leverage this structure and present a novel Motion Transformation Variational Auto-Encoders (MT-VAE) for learning motion sequence generation. Our model jointly learns a feature embedding for motion modes (that the motion sequence can be reconstructed from) and a feature transformation that represents the transition of one motion mode to the next motion mode. Our model is able to generate multiple diverse and plausible motion sequences in the future from the same input. We apply our approach to both facial and full body motion, and demonstrate applications like analogy-based motion transfer and video synthesis. * Work partially done during internship with Adobe Research.

“…Other works introduce recurrent networks into video generation (e.g. [27,28]). In line with these works, our method separately models the motion and appearance as well, using the PSGAN and SCGAN respectively.…”

Section: Pose Sequencesmentioning

confidence: 99%

Pose Guided Human Video Generation

Yang

Wang

Zhu

et al. 2018

117

Due to the emergence of Generative Adversarial Networks, video synthesis has witnessed exceptional breakthroughs. However, existing methods lack a proper representation to explicitly control the dynamics in videos. Human pose, on the other hand, can represent motion patterns intrinsically and interpretably, and impose the geometric constraints regardless of appearance. In this paper, we propose a pose guided method to synthesize human videos in a disentangled way: plausible motion prediction and coherent appearance generation. In the first stage, a Pose Sequence Generative Adversarial Network (PSGAN) learns in an adversarial manner to yield pose sequences conditioned on the class label. In the second stage, a Semantic Consistent Generative Adversarial Network (SCGAN) generates video frames from the poses while preserving coherent appearances in the input image. By enforcing semantic consistency between the generated and ground-truth poses at a high feature level, our SCGAN is robust to noisy or abnormal poses. Extensive experiments on both human action and human face datasets manifest the superiority of the proposed method over other state-of-the-arts.