TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models

Galatolo, Federico A.; Cimino, Mario G. C. A.; Edoardo, Cogotti,

doi:10.48550/arxiv.2212.07839

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Critical to evaluating this is the use of benchmark datasets, i.e., datasets that have previously been generated and have canonical truth identified. Several datasets exist for evaluation AIGC, such as TeTIm-Eval (Text-to-Image Evaluation), which was compared on DALL-E 2, Latent Diffusion, Stable Diffusion, GLIDE (Guided Language to Image Diffusion for Generation and Editing), and Craiyon [84]. Another dataset is the AGIQA-3K dataset (AI-generated Images Quality Assessment-3000), which aims to better capture both human perception and alignment following the Inception Score [85,86].…”

Section: Image Quality Assessmentmentioning

confidence: 99%

Uncertainty in Visual Generative AI

Combs,

Moyer,

Bihl

2024

Algorithms

View full text Add to dashboard Cite

Recently, generative artificial intelligence (GAI) has impressed the world with its ability to create text, images, and videos. However, there are still areas in which GAI produces undesirable or unintended results due to being “uncertain”. Before wider use of AI-generated content, it is important to identify concepts where GAI is uncertain to ensure the usage thereof is ethical and to direct efforts for improvement. This study proposes a general pipeline to automatically quantify uncertainty within GAI. To measure uncertainty, the textual prompt to a text-to-image model is compared to captions supplied by four image-to-text models (GIT, BLIP, BLIP-2, and InstructBLIP). Its evaluation is based on machine translation metrics (BLEU, ROUGE, METEOR, and SPICE) and word embedding’s cosine similarity (Word2Vec, GloVe, FastText, DistilRoBERTa, MiniLM-6, and MiniLM-12). The generative AI models performed consistently across the metrics; however, the vector space models yielded the highest average similarity, close to 80%, which suggests more ideal and “certain” results. Suggested future work includes identifying metrics that best align with a human baseline to ensure quality and consideration for more GAI models. The work within can be used to automatically identify concepts in which GAI is “uncertain” to drive research aimed at increasing confidence in these areas.

show abstract

Section: Image Quality Assessmentmentioning

confidence: 99%

Uncertainty in Visual Generative AI

Combs,

Moyer,

Bihl

2024

Algorithms

View full text Add to dashboard Cite

show abstract

“…There are many models that generate images from text. Galatolo et al constructed a dataset to evaluate image generation models (Galatolo, Cimino, & Cogotti, 2022) In this study, the accuracy of the correspondence between the generated images and the text was evaluated, and DALLE2 (Ramesh, Dhariwal, Nichol, Chu, & Chen, 2022) showed high accuracy following Stable Diffusion (Rombach, Blattmann, Lorenz, Esser, & Ommer, 2022). Brusseau raises concerns about the inappropriate behavior that exists in unconstrained models among these generative models, and positively states the need to regulate them through appropriate controls (Brusseau, 2023).…”

Section: Image Generation Taskmentioning

confidence: 99%

L-VEIGe: Development of L2 Vocabulary Learning Support System with Suggestibility of Error by Image Generation

Sugita,

Gu,

Ota

et al. 2024

Preprint

View full text Add to dashboard Cite

Vocabulary learning has long been taught as a basis for L2 English language learning. The use of visual clues is a widely recognized method in vocabulary learning. Several research have applied automatic image caption generation, invented with computer vision and natural language processing, to English vocabulary learning. However, these vocabulary learning systems mainly use correct answers and their corresponding images. On the other hand, in English vocabulary learning, Fossilization is a problem where errors become difficult to correct and become established through repeated errors by learners. While the use of images has been effective in traditional English vocabulary learning, there has been little research focusing on learners' incorrect answers. In this research, we intentionally created situations where learners are likely to give incorrect answers and constructed an English vocabulary learning support system L-VEIGe (Learning-Vocabulary Error Image Generation) that generates images in response to learners' incorrect answers to promote effective introspection and eliminate repeated errors. An evaluation experiment targeting graduate students who are second language English learners revealed that a proposed method effectively prevents repetitive errors compared to a method without image generation.

show abstract

Speculative Design with Generative AI: Applying Stable Diffusion and ChatGPT to imagining climate change futures

LC,

Tang

2023

Proceedings of the 11th International Conference on Digital and Interactive Arts

View full text Add to dashboard Cite

TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models

Cited by 3 publications

References 0 publications

Uncertainty in Visual Generative AI

Uncertainty in Visual Generative AI

L-VEIGe: Development of L2 Vocabulary Learning Support System with Suggestibility of Error by Image Generation

Speculative Design with Generative AI: Applying Stable Diffusion and ChatGPT to imagining climate change futures

Contact Info

Product

Resources

About