UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

Abstract

An epic dreamscape with fantasy architecture, vivid colors, wide angle, super highly detailed, professional digital painting Legendary elegant gnome hold map and feel confuse in forest, highly detailed, global illumination, ray tracing, sharp focus Beautiful village around an ancient dragon head, massive scale, realistic concept art, cinematic color scheme, dramatic lighting (a) Samples of complex scene image generation (b) Samples of simple scene image generation Beautiful robot female with closed eyes, sci-fi, fantasy, mythology, complex, elegant, highly detailed, digital painting A portrait of a pink cat with human blue eyes wearing a scarf and a top hat, high quality, painting Full body portrait of a squirrel girl, blush, cute and elegant, with squirrel tail cocked to the left

Generative Adversarial Networks for Diverse and Explainable Text-to-Image Generation

“…There are also other metrics for text-to-image evaluation, including Inception score (IS) [85] for image quality and R-precision for text-to-image generation [9]. [12] 27.10 LAFITE [60] 26.94 DALLE [11] 17.89 GLIDE [15] 12.24 Imagen [16] 7.27 Stable Diffusion [17] 12.63 VQ-Diffussion [43] 13.86 DALL-E 2 [18] 10.39 Upainting [63] 8.34 ERNIE-ViLG 2.0 [64] 6.75 eDiff-I [65] 6.95…”

“…Recent evaluation benchmarks. Apart from the automatic metrics discussed above, multiple works involve human evaluation and propose their new evaluation benchmarks [14], [16], [63], [73], [80], [86], [87]. We summarize representative benchmarks in Table 2.…”

“…Specifically, GLIDE [15] finds that CLIP-guidance underperforms the classifier-free variant of guidance. By contrast, another work UPainting [63] points out that lacking of a large-scale transformer language model makes these models with CLIP guidance difficult to encode text prompts and generate complex scenes with details. By combing large language model and cross-modal matching models, UPainting [63] significantly improves the sample fidelity and image-text alignment of generated images.…”

“…By contrast, another work UPainting [63] points out that lacking of a large-scale transformer language model makes these models with CLIP guidance difficult to encode text prompts and generate complex scenes with details. By combing large language model and cross-modal matching models, UPainting [63] significantly improves the sample fidelity and image-text alignment of generated images. The general image synthesis capability enables UPainting [63] to generate images in both simple and complex scenes.…”

“…By combing large language model and cross-modal matching models, UPainting [63] significantly improves the sample fidelity and image-text alignment of generated images. The general image synthesis capability enables UPainting [63] to generate images in both simple and complex scenes.…”