Structural Analogy from a Single Image Pair

Benaim, Sagie; Mokady, Ron; Bermano, Amit H.; Wolf, Lior

doi:10.1111/cgf.14186

Cited by 20 publications

(22 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Single-image generative models have been explored for domains other than textures. GANs trained on a single image have been used for image retargeting [61], deep image analogies [62], or for learning a single-sample image generative model [63], [64], [65], [66], [67]. These methods, while powerful for natural images, are not well-behaved for textures, as shown in [15] and [18].…”

Section: Deep Internal Learningmentioning

confidence: 99%

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Rodriguez-Pardo,

Garces

2022

Preprint

View full text Add to dashboard Cite

Real-time graphics applications require high-quality textured materials to convey realism in virtual environments. Generating these textures is challenging as they need to be visually realistic, seamlessly tileable, and have a small impact on the memory consumption of the application. For this reason, they are often created manually by skilled artists. In this work, we present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar. In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously. Our key idea is to realize that tiling a latent space within a generative network trained using adversarial expansion techniques produces outputs with continuity at the seam intersection that can then be turned into tileable images by cropping the central area. Since not every value of the latent space is valid to produce high-quality outputs, we leverage the discriminator as a perceptual error metric capable of identifying artifact-free textures during a sampling process. Further, in contrast to previous work on deep texture synthesis, our model is designed and optimized to work with multi-layered texture representations, enabling textures composed of multiple maps such as albedo, normals, etc. We extensively test our design choices for the network architecture, loss function, and sampling parameters. We show qualitatively and quantitatively that our approach outperforms previous methods and works for textures of different types.

show abstract

Section: Deep Internal Learningmentioning

confidence: 99%

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Rodriguez-Pardo,

Garces

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Exploiting the power of deep neural networks in the image analogies problem was tackled by Liao et al [23] who, by assuming a semantic prior over an exemplar input image and a target one, propose a method capable of finding a bijective mapping between both inputs, enabling twoway stylizations. Single-image generative models [24] were extended to the image analogies problem in [25], by using convolutional neural networks to generate a new image with the style of an input style and the structure of another image. These methods, however, rely heavily on content or semantic features, making them vulnerable to lighting or geometric differences between the input images; and are computationally expensive, rendering them impractical for interactive applications.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, our model is trained in less time with a smaller computational footprint (1 minute vs 5 minutes). [23] and Structural Analogies [25].…”

Section: Interactive Stylizationsmentioning

confidence: 99%

“…Using our single diffuse image for input, we compare our approach with two methods as shown in Figure 10. First, the work of Liao et al [23], that uses deep latent spaces as image descriptors; and the method of Benaim et al [25] that trains single-image generative models to find bijective mappings between the structure of one image and the style of another. Our method qualitatively outperforms these methods with a fraction of the computational cost: 1 minute in our case, 10 minutes in [3], 40 minutes in [23] and 10 hours in [25].…”

Section: Interactive Stylizationsmentioning

confidence: 99%

“…First, the work of Liao et al [23], that uses deep latent spaces as image descriptors; and the method of Benaim et al [25] that trains single-image generative models to find bijective mappings between the structure of one image and the style of another. Our method qualitatively outperforms these methods with a fraction of the computational cost: 1 minute in our case, 10 minutes in [3], 40 minutes in [23] and 10 hours in [25]. Once trained, our models can be used to evaluate any guidance image in real time for materials with similar microstructure.…”

Section: Interactive Stylizationsmentioning

confidence: 99%

See 2 more Smart Citations

Neural Photometry-guided Visual Attribute Transfer

Rodriguez-Pardo,

Garces

2021

Preprint

View full text Add to dashboard Cite

We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g. texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following an image analogies approach, the method only requires the training data to contain the same visual structures as the input guidance. Our approach works at interactive rates, making it suitable for material edit applications. We thoroughly evaluate our learning methodology in a controlled setup providing quantitative measures of performance. Last, we demonstrate that training the model on a single material is enough to generalize to materials of the same type without the need for massive datasets.

show abstract