Lagrangian neural style transfer for fluids

Abstract

Artistically controlling the shape, motion and appearance of fluid simulations pose major challenges in visual effects production. In this paper, we present a neural style transfer approach from images to 3D fluids formulated in a Lagrangian viewpoint. Using particles for style transfer has unique benefits compared to grid-based techniques. Attributes are stored on the particles and hence are trivially transported by the particle motion. This intrinsically ensures temporal consistency of the optimized stylized structure and notably improves the resulting quality. Simultaneously, the expensive, recursive alignment of stylization velocity fields of grid approaches is unnecessary, reducing the computation time to less than an hour and rendering neural flow stylization practical in production settings. Moreover, the Lagrangian representation improves artistic control as it allows for multi-fluid stylization and consistent color transfer from images, and the generality of the method enables stylization of smoke and liquids likewise.

Accelerated Smoke Simulation by Super-Resolution With Deep Learning on Downscaled and Binarized Space

“…Because of the elimination of unnecessary SR operations, the performance of our method is significantly improved compared with using the original tempoGAN [28]. Because the space partitioning technique in this study is widely used, it can be easily applied when performing SR operations for other types of fluid simulation techniques (e.g., Sato et al [43] and Kim et al [5], [6]).…”

“…In future work, we aim to investigate the automatic determination of suitable patch sizes, which would improve the efficiency of our method. Because our method of partitioning the simulation space to be operated on is a generalizable process, our approach should be applicable to the acceleration of other fluid-simulation-based SR models or style-transfer models (e.g., Sato et al [43] and Kim et al [5], [6]).…”

“…Our method improves the efficiency in terms of speed and memory by applying the SR operation to the smoke simulation based on the adaptive grid (e.g., octree). Although our method does not represent realistic elements of the smoke movement, integrating our method into existing detail-enhancement studies can improve details and efficiency [5], [6], [51]. Some studies improve detail using deep learning for fluid simulation.…”

“…B ECAUSE of advances in deep learning in recent years, physics-based simulation fields such as character animations [13], [14], [29] and fluid simulations [6], [15], [24], [30] have also been remarkably improved because of deep learning. Accuracy and efficiency in fields such as style transfer [5], [16], [27], character motion control [17], [26], and numerical analysis [18] have also improved. However, in certain fields, it still suffers from the need for a large amount of computation.…”

Patch Erosion for Deformable Lapped Textures on 3D Fluids

“…Neural networks have been used for style transfer on 2D smoke simulation [CKAS20] and later extended on volumetric fluid animation, changing also the shape of 3D fluid surfaces [KAGS20]. Where these methods focus on volumetric data, our approach concentrates on surface texture synthesis.…”