Adversarial Generation of Continuous Images

Skorokhodov, Ivan; Ignatyev, Savva; Elhoseiny, Mohamed

doi:10.1109/cvpr46437.2021.01061

Cited by 88 publications

(60 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The largest popularity of implicit neural representations/functions (INRs) is studied in 3D deep learning to represent a 3D shape in a cheap and continuous way [35,36,37]. Recent studies [1,10,29,50] explore the idea of using INRs for image generation, where they learn a hyper MLP network to predict an RGB pixel value given its coordinates on the image grid. Among them, [1,50] are closely related to our high-resolution stages of the generative process.…”

Section: Related Workmentioning

confidence: 99%

“…Recent studies [1,10,29,50] explore the idea of using INRs for image generation, where they learn a hyper MLP network to predict an RGB pixel value given its coordinates on the image grid. Among them, [1,50] are closely related to our high-resolution stages of the generative process. One remarkable difference is that our model is driven by the cross-attention module and features generated in previous stages instead of the hyper-network presented in [1,50].…”

Section: Related Workmentioning

confidence: 99%

“…Among them, [1,50] are closely related to our high-resolution stages of the generative process. One remarkable difference is that our model is driven by the cross-attention module and features generated in previous stages instead of the hyper-network presented in [1,50].…”

Section: Related Workmentioning

confidence: 99%

“…datasets. On the FFHQ dataset, apart from our ConvNet-based counterparts, we also compare to the most recent INR-based methods including CIPS [1] and INR-GAN [50]. These two models are closely related to HiT as the high-resolution stages of our approach can be viewed as a form of INR.…”

Section: Higher Resolution Generationmentioning

confidence: 99%

See 3 more Smart Citations

Improved Transformer for High-Resolution GANs

Zhao

Zhang

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

Attention-based models, exemplified by the Transformer, can effectively model long range dependency, but suffer from the quadratic complexity of self-attention operation, making them difficult to be adopted for high-resolution image generation based on Generative Adversarial Networks (GANs). In this paper, we introduce two key ingredients to Transformer to address this challenge. First, in low-resolution stages of the generative process, standard global self-attention is replaced with the proposed multi-axis blocked self-attention which allows efficient mixing of local and global attention. Second, in high-resolution stages, we drop self-attention while only keeping multi-layer perceptrons reminiscent of the implicit neural function. To further improve the performance, we introduce an additional selfmodulation component based on cross-attention. The resulting model, denoted as HiT, has a linear computational complexity with respect to the image size and thus directly scales to synthesizing high definition images. We show in the experiments that the proposed HiT achieves state-of-the-art FID scores of 31.87 and 2.95 on unconditional ImageNet 128 × 128 and FFHQ 256 × 256, respectively, with a reasonable throughput. We believe the proposed HiT is an important milestone for generators in GANs which are completely free of convolutions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Higher Resolution Generationmentioning

confidence: 99%

See 2 more Smart Citations

Improved Transformer for High-Resolution GANs

Zhao

Zhang

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Several works utilize INRs for generative modeling, i.e., samples are generated through INRs (Chan et al, 2021;Dupont et al, 2021b;Kosiorek et al, 2021). In particular, Skorokhodov et al (2021a) and Anokhin et al (2021) exposed that INR-based image generative adversarial networks (GANs; Goodfellow et al (2014)), which generate images as INRs, show impressive generation performance. Interestingly, they further merit various advantages of INRs, e.g., natural inter-and extra-polation, anycost inference (i.e., control the trade-off of quality and cost), and parallel computation, which needs a non-trivial modification to apply under other generative model architectures.…”

Section: Introductionmentioning

confidence: 99%

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Yu¹,

Tack²,

Mo³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the deep learning era, long video generation of high-quality still remains challenging due to the spatio-temporal complexity and continuity of videos. Existing prior works have attempted to model video distribution by representing videos as 3D grids of RGB values, which impedes the scale of generated videos and neglects continuous dynamics. In this paper, we found that the recent emerging paradigm of implicit neural representations (INRs) that encodes a continuous signal into a parameterized neural network effectively mitigates the issue. By utilizing INRs of video, we propose dynamics-aware implicit generative adversarial network (DI-GAN), a novel generative adversarial network for video generation. Specifically, we introduce (a) an INR-based video generator that improves the motion dynamics by manipulating the space and time coordinates differently and (b) a motion discriminator that efficiently identifies the unnatural motions without observing the entire long frame sequences. We demonstrate the superiority of DIGAN under various datasets, along with multiple intriguing properties, e.g., long video synthesis, video extrapolation, and non-autoregressive video generation. For example, DIGAN improves the previous state-of-the-art FVD score on UCF-101 by 30.7% and can be trained on 128 frame videos of 128×128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method. 1

show abstract