Real-time Localized Photorealistic Video Style Transfer

Xia, Xide; Xue, Tao; Lai, Wei-Sheng; Sun, Zheng; Chang, Abby; Kulis, Brian; Chen, Jiawen

doi:10.1109/wacv48630.2021.00113

Cited by 24 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works adopt particular style transfer operators [17,19,28,29,37,59] to the Encoder-Decoder pre-trained on natural image dataset, e.g., COCO [33]. The advances [60,61] learn the edge-preserved local affine grid [12] in bilateral space [5] to transfer the color locally. Because the input size and semantic mask heavily influence these methods, directly applying to 3D scene will produce distortion and inconsistency.…”

Section: Style Transfer Methodsmentioning

confidence: 99%

Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization

Zhang¹,

Liu²,

Han³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advances in 3D scene representation and novel view synthesis have witnessed the rise of Neural Radiance Fields (NeRFs). Nevertheless, it is not trivial to exploit NeRF for the photorealistic 3D scene stylization task, which aims to generate visually consistent and photorealistic stylized scenes from novel views. Simply coupling NeRF with photorealistic style transfer (PST) will result in cross-view inconsistency and degradation of stylized view syntheses. Through a thorough analysis, we demonstrate that this nontrivial task can be simplified in a new light: When transforming the appearance representation of a pre-trained NeRF with Lipschitz mapping, the consistency and photorealism across source views will be seamlessly encoded into the syntheses. That motivates us to build a concise and flexible learning framework namely LipRF, which upgrades arbitrary 2D PST methods with Lipschitz mapping tailored for the 3D scene. Technically, LipRF first pre-trains a radiance field to reconstruct the 3D scene, and then emulates the style on each view by 2D PST as the prior to learn a Lipschitz network to stylize the pre-trained appearance. In view of that Lipschitz condition highly impacts the expressivity of the neural network, we devise an adaptive regularization to balance the reconstruction and stylization. A gradual gradient aggregation strategy is further introduced to optimize LipRF in a cost-efficient manner. We conduct extensive experiments to show the high quality and robust performance of LipRF on both photorealistic 3D stylization and object appearance editing.

show abstract

Section: Style Transfer Methodsmentioning

confidence: 99%

Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization

Zhang¹,

Liu²,

Han³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For blind approaches without dedicated guiding style reference, this can be done using optical flow to calculate temporal losses [Chen et al , 2020 or align intermediate feature representations [Gao et al 2018; to stabilize models' prediction across nearby video frames. Recently, there have been efforts to improve consistency and speed for video style transfer for arbitrary styles through temporal regularization [Wang et al 2020a], multi-channel correlation , and bilateral learning [Xia et al 2021]. Similarly, style transfer for stereo images [Chen et al 2018;Gong et al 2018] also aims to achieve cross-view consistency by using dense pixel correspondences (via stereo matching) constraints.…”

Section: Related Work 21 Image and Video Style Transfermentioning

confidence: 99%

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

Nguyen-Phuoc¹,

Liu²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

Fig. 1. Given a neural implicit scene representation trained with multiple views of a scene, SNeRF stylizes the 3D scene to match a reference style. SNeRF works with a variety of scene types (indoor, outdoor, 4D dynamic avatar) and generates novel views with cross-view consistency.This paper presents a stylized novel view synthesis method. Applying stateof-the-art stylization methods to novel views frame by frame often causes jittering artifacts due to the lack of cross-view consistency. Therefore, this paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. Specifically, we adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation for their capability to render high-quality novel views for a variety of scenes. However, as rendering a novel view from a NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that goes beyond an off-the-shelf GPU capacity. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps. Such a method enables us to make full use of our hardware memory capacity to both generate images at higher resolution and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor and dynamic scenes, and synthesizes high-quality novel views with cross-view consistency.

show abstract

“…The latest techniques of neural style transfers can achieve real-time processing of videos. The videos of virtual or physical worlds can be alternated by multitudinous user-defined styles [96] (Figure 7a), which achieve high reasonable granularity of processing the video scenes by providing multiple styles for different object classes [214] (Figure 7b). This implies that metaverse creators can choose their preferred styles, and offer highly personalised yet artistic experiences to other metaverse individuals, due to the nature of cinema-alike metaverse.…”

Section: Virtual Photography / Cinematic Simulationmentioning

confidence: 99%

When Creators Meet the Metaverse: A Survey on Computational Arts

Lee

Lin²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The metaverse, enormous virtual-physical cyberspace, has brought unprecedented opportunities for artists to blend every corner of our physical surroundings with digital creativity. This article conducts a comprehensive survey on computational arts, in which seven critical topics are relevant to the metaverse, describing novel artworks in blended virtual-physical realities. The topics first cover the building elements for the metaverse, e.g., virtual scenes and characters, auditory, textual elements. Next, several remarkable types of novel creations in the expanded horizons of metaverse cyberspace have been reflected, such as immersive arts, robotic arts, and other user-centric approaches fuelling contemporary creative outputs. Finally, we propose several research agendas: democratising computational arts, digital privacy and safety for metaverse artists, ownership recognition for digital artworks, technological challenges, and so on. The survey also serves as introductory material for artists and metaverse technologists to begin creations in the realm of surrealistic cyberspace.CCS Concepts: • Human-centered computing → Mixed / augmented reality; Graphical user interfaces; • Computing methodologies → Machine learning approaches.

show abstract

Real-time Localized Photorealistic Video Style Transfer

Cited by 24 publications

References 29 publications

Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization

Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

When Creators Meet the Metaverse: A Survey on Computational Arts

Contact Info

Product

Resources

About