StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions

Höllein, Lukas; Johnson, Justin C.; Nießner, Matthias

doi:10.48550/arxiv.2112.01530

Cited by 2 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, we move on to the comparison with the approaches of Gutierrez et al [GRGH19a], Mordvintsev et al [MPSO18a], and Höllein et al [HJN22a]. The approach of Gutierrez et al works particularly well for isotropic materials, such as the marble texture.…”

Section: Comparison To the State Of The Artmentioning

confidence: 99%

“…To address these shortcomings, render-based approaches adopt a process of rendering the given mesh and subsequently matching the style of the projected parts [KUH17,MPSO18a,HJN22a]. Such methods require rendering the object from multiple viewpoints, thereby, introducing challenges when dealing with objects of high complexity, with multiple holes, or overlapping wire-like features, where achieving visibility of all surfaces becomes difficult.…”

Section: Image Synthesismentioning

confidence: 99%

“…This is a technique where a texture is progressively downsampled to increase rendering speed and reduce aliasing artifacts. In neural approaches, mipmapping can be used as a pooling operation [LLZ * 19] or as a method for working with coarser fea-tures [HJN22a]. However, this approach may introduce errors.…”

Section: Comparison To the State Of The Artmentioning

confidence: 99%

“…However, these methods are designed to generate highfrequency and abstract textures, such as marble, and do not consider the lower-dimensional manifold of 3D meshes during the generation process. Another line of research employs 2D CNNs in conjunction with a differentiable renderer to optimize the resulting mesh texture from a given 2D exemplar across multiple viewpoints [HJN22a,MPSO18a]. These methods heavily rely on sampling a large number of views, which can be computationally demanding and may lead to occlusion-related inconsistencies.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs

Kovács,

Hermosilla,

Raidou

2024

Computer Graphics Forum

View full text Add to dashboard Cite

Mesh texture synthesis is a key component in the automatic generation of 3D content. Existing learning‐based methods have drawbacks—either by disregarding the shape manifold during texture generation or by requiring a large number of different views to mitigate occlusion‐related inconsistencies. In this paper, we present a novel surface‐aware approach for mesh texture synthesis that overcomes these drawbacks by leveraging the pre‐trained weights of 2D Convolutional Neural Networks (CNNs) with the same architecture, but with convolutions designed for 3D meshes. Our proposed network keeps track of the oriented patches surrounding each texel, enabling seamless texture synthesis and retaining local similarity to classical 2D convolutions with square kernels. Our approach allows us to synthesize textures that account for the geometric content of mesh surfaces, eliminating discontinuities and achieving comparable quality to 2D image synthesis algorithms. We compare our approach with state‐of‐the‐art methods where, through qualitative and quantitative evaluations, we demonstrate that our approach is more effective for a variety of meshes and styles, while also producing visually appealing and consistent textures on meshes.

show abstract

Section: Comparison To the State Of The Artmentioning

confidence: 99%

Section: Image Synthesismentioning

confidence: 99%

Section: Comparison To the State Of The Artmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs

Kovács,

Hermosilla,

Raidou

2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…For style transfer at 3D scene level, recent approaches use point clouds [Cao et al 2020;Kopanas et al 2021] or meshes [Höllein et al 2021] as scene representations. However, these approaches are limited to static scenes.…”

Section: D Style Transfermentioning

confidence: 99%

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

Nguyen-Phuoc¹,

Liu²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

Fig. 1. Given a neural implicit scene representation trained with multiple views of a scene, SNeRF stylizes the 3D scene to match a reference style. SNeRF works with a variety of scene types (indoor, outdoor, 4D dynamic avatar) and generates novel views with cross-view consistency.This paper presents a stylized novel view synthesis method. Applying stateof-the-art stylization methods to novel views frame by frame often causes jittering artifacts due to the lack of cross-view consistency. Therefore, this paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. Specifically, we adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation for their capability to render high-quality novel views for a variety of scenes. However, as rendering a novel view from a NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that goes beyond an off-the-shelf GPU capacity. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps. Such a method enables us to make full use of our hardware memory capacity to both generate images at higher resolution and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor and dynamic scenes, and synthesizes high-quality novel views with cross-view consistency.

show abstract

StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions

Cited by 2 publications

References 43 publications

Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs

Surface‐aware Mesh Texture Synthesis with Pre‐trained 2D CNNs

SNeRF: Stylized Neural Implicit Representations for 3D Scenes

Contact Info

Product

Resources

About