Semantic Pyramid for Image Generation

Shocher, Assaf; Gandelsman, Yossi; Mosseri, Inbar; Yarom, Michal; Irani, Michal; Freeman, William T.; Dekel, Tali

doi:10.1109/cvpr42600.2020.00748

Cited by 63 publications

(33 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Characterizing such features has been a key question in scene perception for decades (Davenport & Potter, 2004;Greene & Oliva, 2009;Kauffmann et al, 2015;Oliva & Schyns, 2000;Oliva & Torralba, 2001;Schyns & Oliva, 1994;Walther et al, 2011); however, the recent revolution in deep learning approaches from machine learning and computer vision have provided novel insights into the nature of features underlying visual categorization (Cichy et al, 2017;Eberhardt et al, 2016;Krizhevsky et al, 2012;Rezanejad et al, 2019;Zhou et al, 2017;for review, Serre, 2019). In particular, generative adversarial networks (GAN; Brock et al, 2018;Goodfellow et al, 2014;Karras et al, 2019;Shocher et al, 2020;Yang et al, 2019) offer a data-driven method for uncovering the complex features spaces necessary for generating artificial but highly realistic images. GANs are composed of two deep neural networks, a generative network and a discriminative network, which perform antagonistic tasks.…”

mentioning

confidence: 99%

“…After successful learning, the synthesized images created by the generative network are highly realistic even to human observers. Recent studies of scene specific GANs (Shocher et al, 2020;Yang et al, 8 2019) have demonstrated that latent spaces from trained generative networks show a meaningful organization such that adjacent vectors in latent space are highly similar to each other and their associated synthetic images represent continuously changing realistic scenes. These advances offer a unique possibility -a continuous space of natural scene images may be constructed by deliberately traversing the latent space of a GAN trained to synthesize scene images.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space

Son

Walther

Mack

2020

Preprint

View full text Add to dashboard Cite

Precisely characterizing mental representations of visual experiences requires careful control of experimental stimuli. Recent work leveraging such stimulus control in continuous report paradigms have led to important insights; however, these findings are constrained to simple visual properties like colour and line orientation. There remains a critical methodological barrier to characterizing perceptual and mnemonic representations of realistic visual experiences. Here, we introduce a novel method to systematically control visual properties of natural scene stimuli. Using generative adversarial networks (GAN), a state-of-art deep learning technique for creating highly realistic synthetic images, we generated scene wheels in which continuously changing visual properties smoothly transition between meaningful realistic scenes. To validate the efficacy of scene wheels, we conducted a memory experiment in which participants reconstructed to-be-remembered scenes from the scene wheels. Reconstruction errors for these scenes resemble error distributions observed in prior studies using simple stimulus properties. Importantly, memory precision varied systematically with scene wheel radius. These findings suggest our novel approach offers a window into the mental representations of naturalistic visual experiences.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space

Son

Walther

Mack

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Initially, linear normalization [17] was applied on the dataset as a pre-processing stage before applying the image registration processes. Then, the Gaussian image pyramid [18,19] was used for down sampling to reduce processing time and memory size.…”

Section: Methodsmentioning

confidence: 99%

Nonrigid Medical Image Registration using Adaptive Gradient Optimizer

Arab

El-Khobby

Ashour³

2021

Journal of Engineering Research

View full text Add to dashboard Cite

Medical image registration has a significant role in several applications. It has sequential processes, including transformation, similarity metric calculation, diffusion regularization, and optimization of the transformation parameters (i.e., rotation, translation, and shear). The optimization process for determining the optimal set of the transformation vectors is considered the main stage affecting the performance of the registration process. Hence, medical image registration can be deliberated as an optimization problem for computing the geometric transformations to realize maximum similarity between the moving image and the fixed one. In this paper a mono-modal nonrigid image registration using B-spline is designed for the alignment of Computed Tomography (CT) images using Adaptive Gradient algorithm (AdaGrad) optimizer. In addition, a comparative study with other first order optimizers, such as Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam) algorithm (AdaMaX), AdamP, and RangerQH were conducted. Also, a comparison with the limited memory Broyden-Fletcher-Goldfarb-Shannon (LBFGS) as a second order optimizer was also carried out. The results showed the superiority of the AdaGrad optimizer by 56.99% and 48.37% improvement in the target registration error (TRE) compared to the SGD, and the LBFGS optimizer, respectively.

show abstract

“…The image pyramid [26] is a collection of images at different scales as multi-scale representations of images. The semantic pyramid [27] is built on the basis of the image pyramid, and the model is constructed as a semantic pyramidal generation: the low-level information contains fine features (texture details, etc. ), while the high-level information overlays high-level semantic information.…”

Section: Semantic Pyramid Networkmentioning

confidence: 99%

“…Moreover, the residual blocks are added to each layer of the pyramid network to address the problem of reduced accuracy caused by the increase in the depth of the neural network. Therefore, we present the residual pyramid encoder built on the semantic pyramid structure [27]. It encodes the damaged images into compact latent features and decodes these features back to the image.…”

Section: Residual Pyramid Encodermentioning

confidence: 99%

Semantic Residual Pyramid Network for Image Inpainting

Luo

Zheng

2022

Information

View full text Add to dashboard Cite

Existing image inpainting methods based on deep learning have made great progress. These methods either generate contextually semantically consistent images or visually excellent images, ignoring that both semantic and visual effects should be appreciated. In this article, we propose a Semantic Residual Pyramid Network (SRPNet) based on a deep generative model for image inpainting at the image and feature levels. This method encodes a masked image by a residual semantic pyramid encoder and then decodes the encoded features into a inpainted image by a multi-layer decoder. At this stage, a multi-layer attention transfer network is used to gradually fill in the missing regions of the image. To generate semantically consistent and visually superior images, the multi-scale discriminators are added to the network structure. The discriminators are divided into global and local discriminators, where the global discriminator is used to identify the global consistency of the inpainted image, and the local discriminator is used to determine the consistency of the missing regions of the inpainted image. Finally, we conducted experiments on four different datasets. As a result, great performance was achieved for filling both the regular and irregular missing regions.

show abstract

Semantic Pyramid for Image Generation

Cited by 63 publications

References 14 publications

Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space

Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space

Nonrigid Medical Image Registration using Adaptive Gradient Optimizer

Semantic Residual Pyramid Network for Image Inpainting

Contact Info

Product

Resources

About