Contrastive language and vision learning of general fashion concepts

Abstract

The steady rise of online shopping goes hand in hand with the development of increasingly complex ML and NLP models. While most use cases are cast as specialized supervised learning problems, we argue that practitioners would greatly benefit from general and transferable representations of products. In this work, we build on recent developments in contrastive learning to train FashionCLIP, a CLIP-like model adapted for the fashion industry. We demonstrate the effectiveness of the representations learned by FashionCLIP with extensive tests across a variety of tasks, datasets and generalization probes. We argue that adaptations of large pre-trained models such as CLIP offer new perspectives in terms of scalability and sustainability for certain types of players in the industry. Finally, we detail the costs and environmental impact of training, and release the model weights and code as open source contribution to the community.

Auditing Gender Presentation Differences in Text-to-Image Models

“…image models with occupations and human-related objects and analyze the generated people's genders and skin colors. Bianchi et al (2022) further observe that the stereotypes related to professions in generated images are amplified compared to the real-world distributions of occupations. For example, although women nurses are in the majority in real life, 45 nearly all nurses generated by the model are women, which is extremely imbalanced.…”

“…• Unlike the demographic information related to occupations (Bianchi et al, 2022), we have no access to the real-world distributions of studied attributes. Assuming that attributes are distributed uniformly across genders may not capture real-world scenarios well.…”

“…(OpenAI, 2022). Such stereotypes are amplified in text-to-image models compared to real-world distributions (Bianchi et al, 2022). However, how different social groups are represented unequally in text-to-image models is far from comprehensively studied.…”

“…Taking gender as an example, most prior studies prompt the model with "A photo of [X]." where "[X]" refers to occupations ("a CEO") or descriptors ("an attractive person", "a person with a beer") (Cho et al, 2022;Bianchi et al, 2022;Fraser et al, 2023). These approaches automatically classify all generated images into gender categories and measure bias using the relative gender frequencies.…”

“…Extensive evaluation using human annotation for all these text-to-image models is impractical, as collecting (or reusing) ground truth images for every attribute of interest (Bianchi et al, 2022) is not scalable. In prior works, state-of-the-art textimage matching models like CLIP (Radford et al, 2021) are used for zero-shot gender classification to quantify gender biases.…”