Generating Compositional Color Representations from Text

Abstract

We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition -the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Fréchet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks. CCS CONCEPTS• Information systems → Query intent; Content ranking; Novelty in information retrieval.

Color Recommendation for Vector Graphic Documents based on Multi-Palette Representation

“…Kikuchi et al [10] proposed maximum likelihood estimation (MLE) and conditional variational autoencoder (CVAE) models using the Transformer-based network to recommend text and background colors for each element with text content in ecommerce mobile web pages. Moreover, Maheshwari et al [14] proposed a conditional generative adversarial networks (GAN) architecture for generating a color palette for image colorization with an attribute-object pair text input, such as 'warm sunshine' and 'cute dog'. Likewise, Bahng et al [3] proposed a conditional GAN based text-to-palette generation networks for image colorization that reflect the semantics of text input.…”

“…However, this work only examined the relationships among colors in multiple palettes. Some studies based on multi-modality learning have aimed to generate a color palette based on textual information for image colorization [3,14]. The text in these works comprises a brief sequence, such as a single word (e.g., 'sunny'), an attribute-object pair (e.g., 'cute dog'), or a phrase (e.g., 'grape to strawberry').…”