DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Cho, Jaemin; Zala, Abhay; Bansal, Mohit

doi:10.48550/arxiv.2202.04053

Cited by 18 publications

(34 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For model analysis in Subsection 2.2, we randomly sample 3K prompts from the COCO for efficiency and report both FID and CLIPscore. Besides COCO, several other benchmarks have also been proposed to systematically evaluate the performance of different text-to-image models, such as PaintSkills [Cho et al, 2022], DrawBench [Saharia et al, 2022] and PartiPrompts [Yu et al, 2022]. These benchmarks don't have reference images, so it's difficult to make automatic evaluation on them.…”

Section: Evaluation Benchmarkmentioning

confidence: 99%

UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

Li¹,

Xue²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

An epic dreamscape with fantasy architecture, vivid colors, wide angle, super highly detailed, professional digital painting Legendary elegant gnome hold map and feel confuse in forest, highly detailed, global illumination, ray tracing, sharp focus Beautiful village around an ancient dragon head, massive scale, realistic concept art, cinematic color scheme, dramatic lighting (a) Samples of complex scene image generation (b) Samples of simple scene image generation Beautiful robot female with closed eyes, sci-fi, fantasy, mythology, complex, elegant, highly detailed, digital painting A portrait of a pink cat with human blue eyes wearing a scarf and a top hat, high quality, painting Full body portrait of a squirrel girl, blush, cute and elegant, with squirrel tail cocked to the left

show abstract

Section: Evaluation Benchmarkmentioning

confidence: 99%

UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

Li¹,

Xue²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Park et al [11] provide the curated splits of the CUB [28,29] and Flowers [30] to assess unseen color and shape compositions in the narrow domains. DALL-Eval [31] proposed a diagnostic dataset PaintSkills to evaluate visual reasoning skills to assess models based on this dataset. Since the dataset is generated from a 3D simulator using limited configurations, the data distributions deviate from other real-world datasets.…”

Section: Metrics For Assessing Text-to-image Generationmentioning

confidence: 99%

“…The runner-up was InfoNCE, while SOA was ineffective to differentiate among the two fake images and the real image. Although DALL-Eval [31] proposed to use a detector and its dedicated heads for the count, color, and spatial relationship tasks, this was limited to the 3D-generated images with their near-perfect detection ability. Remind that the accuracy of random guessing is 33.3%, where MID requires a powerful feature extractor of the CLIP ViT-L/14 to get meaningful performances on the count, color, and spatial relationship tasks.…”

Section: Evaluation On Text-to-image Generationmentioning

confidence: 99%

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

Kim¹,

Kim²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

Text-to-image generation and image captioning are recently emerged as a new experimental paradigm to assess machine intelligence. They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable to get marginal distributions. Based on a recent trend that multimodal generative evaluations exploit a vison-and-language pre-trained model, we propose the negative Gaussian cross-mutual information using the CLIP features as a unified metric, coined by Mutual Information Divergence (MID). To validate, we extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model. We look forward to seeing the underrepresented implications of the Gaussian cross-mutual information in multimodal representation learning and the future works based on this novel proposition.

show abstract

“…This has been further extended to develop natural language explanations by using captioning methods to describe a set of image patches that activated a neuron [107], [188]. Aside from these, dissection has also been used to analyze what types of neurons are exploited by adversarial examples [263], identify failure modes for text-toimage models [49], and probe neural responses in transformers to isolate where certain facts are stored [169]. Unfortunately, these types of methods are limited by the diversity of examples in the dataset used and the quality of labels.…”

Section: B Dataset-based (Post Hoc)mentioning

confidence: 99%

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman¹,

Ho²,

Casper³

et al. 2022

Preprint

View full text Add to dashboard Cite

The last decade of machine learning has seen drastic increases in scale and capabilities, and deep neural networks (DNNs) are increasingly being deployed across a wide range of domains. However, the inner workings of DNNs are generally difficult to understand, raising concerns about the safety of using these systems without a rigorous understanding of how they function. In this survey, we review literature on techniques for interpreting the inner components of DNNs, which we call inner interpretability methods. Specifically, we review methods for interpreting weights, neurons, subnetworks, and latent representations with a focus on how these techniques relate to the goal of designing safer, more trustworthy AI systems. We also highlight connections between interpretability and work in modularity, adversarial robustness, continual learning, network compression, and studying the human visual system. Finally, we discuss key challenges and argue for future work in interpretability for AI safety that focuses on diagnostics, benchmarking, and robustness.

show abstract

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Cited by 18 publications

References 41 publications

UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Contact Info

Product

Resources

About