Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Abstract

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps.Step 1 synthesizes video sequences from the currently learned model.Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns. We also show that it is possible to learn the model from incomplete training sequences with either occluded pixels or missing frames, so that model learning and pattern completion can be accomplished simultaneously.

General Hypernetwork Framework for Creating 3D Point Clouds

“…In such a case, new point clouds can be generated using Monte Carlo Markov Chain (MCMC) sampling. Such an architecture was used to generate images [21], videos [22], [23], [24], and 3D voxels [25], [26]. One of the most recent models, Generative PointNet (GPN), applies this approach to 3D point clouds [27].…”

ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

“…Classic synthesis methods rely on mathematical models such as Markov random fields [56] and auto-regressive moving average model [16] to capture underlying motion characteristics. More recently, deep learning techniques, in particular 3D CNNs and GAN-based training [19,55,58,62,63], have been adopted to achieve more realistic synthesis results. It should be noted that both dynamic texture synthesis and VFI require accurate modeling of spatio-temporal characteristics.…”

“…Including additional frames here allows better modeling of higher-order motions and also provides more information on longer-term spatiotemporal characteristics. Motivated by recent work in dynamic texture synthesis [58,63], where spatio-temporal filtering was found to be effective for generating coherent video textures, we integrate a 3D CNN for texture enhancement. This CNN architecture (shown in Figure 4) is a modified version of the network developed in [29], but with reduced layer widths.…”

“…To further improve the performance for large motions, a Bi-directional Linear Flow Network (BLFNet) is employed to linearly approximate the intermediate flows based on the bi-directional flows between input frames, which are estimated using a coarse-to-fine architecture [53]. In the second stage, inspired by recent work on texture synthesis [58,63], we integrate a 3D CNN, Texture Enhancement Network (TENet) that performs spatial and temporal filtering to capture longer-range dynamics and to predict textural residuals. Finally, we trained our model based on the ST-GAN [63] methodology, which was originally proposed for texture synthesis.…”

Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction

“…Energy-based models: Recent works have shown strong performance of data space EBMs [72,44] in modeling high-dimensional complex dependencies, such as images [97,95,18,11,19], videos [78,79], 3D shapes [75,76], and point clouds [73], and also demonstrated the effectiveness of latent space EBMs [47] in improving the model expressivity for text [48], image [47], and trajectory [49] generation. Our paper also learns a latent space EBM as the prior model but builds the EBM on top of a vision transformer generator for image-conditioned saliency map prediction.…”