Image Transformer

Abstract

Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the selfattention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our super-resolution model fool human observers three times more often than the previous state of the art.

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

“…The method proposed in this paper follows the line of visual synthesis research based on auto-regressive models. Earlier visual auto-regressive models [5,28,39,41,44] performed visual synthesis in a "pixel-by-pixel" manner. However, due to the high computational cost when modeling high-dimensional data, such methods can be applied to lowresolution images or videos only, and are hard to scale up.…”

“…However, the quality of generated visual contents could be harmed due to the limited contexts used in self-attention. [6,28,32] proposed to use local-wise sparse attention in visual synthesis tasks, which allows the models to see more contexts. But these works were for images only.…”

“…Auto-regressive models [33,39,41,45] play an important role in visual synthesis tasks, due to their explicit density modeling and stable training advantages compared with GANs [4,30,37,47]. Earlier visual auto-regressive models, such as PixelCNN [39], PixelRNN [41], Image Transformer [28], iGPT [5], and Video Transformer [44], performed visual synthesis in a "pixel-by-pixel" manner. However, due to their high computational cost on high-dimensional visual data, such methods can be applied to low-resolution images or videos only and are hard to scale up.…”

DA-FDFtNet: Dual Attention Fake Detection Fine-tuning Network to Detect Various AI-Generated Fake Images

“…Wang et al [37] formalized self-attention as a non-local operation to explore the spatialtemporal dependencies' effectiveness in video and image sequences. Parmar et al [38] introduced Image Transformer, applying the self-attention model into an autoregressive model for image generation. Zhang et al [39] proposed SAGAN, which allowed the self-attention-driven and long-range dependency model for learning a better image generation.…”

SAFL: A Self-Attention Scene Text Recognizer with Focal Loss

“…This mechanism allows more computation parallelization with higher performance. In the computer vision domain, some research have leveraged the transformer architecture and showed the effectiveness of some problems [4] [5] Inspired by the transformer network, in this paper, we propose a self-attention based scene text recognizer with focal loss, namely as SAFL. Moreover, to tackle irregular shapes of scene texts, we also exploit a text rectification named Spatial Transformer Network (STN) to enhance the quality of text before passing to the recognition network.…”