Figure 1. New "pets" generated using ConceptLab. Each pair depicts a learned concept that was optimized to be unique and distinct from existing members of the pet category. Our method can generate a variety of novel concepts from a single broad category.
Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion. github.io → Input samples invert −−−−→ "S * " "An oil painting of S * " "App icon of S * " "Elmo sitting in the same pose as S * " "Crochet S * " → Input samples invert −−−−→ "S * " "Painting of two S * fishing on a boat" "A S * backpack" "Banksy art of S * " "A S * themed lunchbox"Preprint. Under review.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.