Learning Type-Aware Embeddings for Fashion Compatibility

Abstract

Outfits in online fashion data are composed of items of many different types (e.g. top, bottom, shoes) that share some stylistic relationship with one another. A representation for building outfits requires a method that can learn both notions of similarity (for example, when two tops are interchangeable) and compatibility (items of possibly different type that can go together in an outfit). This paper presents an approach to learning an image embedding that respects item type, and jointly learns notions of item similarity and compatibility in an end-toend model. To evaluate the learned representation, we crawled 68,306 outfits created by users on the Polyvore website. Our approach obtains 3-5% improvement over the state-of-the-art on outfit compatibility prediction and fill-in-the-blank tasks using our dataset, as well as an established smaller dataset, while supporting a variety of useful queries 1 .

Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network

“…Shih et al [22] propose to make multiple projection points of a query image. Vasileva et al [28] learn different metric spaces for different type combinations. While evaluating the outfit compatibility, these methods take the average of all pairwise compatibility as the output and neglect the relationship between the pairwise compatibility and the overall compatibility.…”

“…This will leads to an improper situation e.g., if a shirt matches a trouser, the trouser then matches a shoe, the consequence is the shoe is forced to match the shirt. Therefore, we use the projected embedding with respect to different fashion type combinations to address the above problems, which refers to [3,28]. A fashion outfit often contains items from different types such as top, bottom and shoe etc.…”

“…Pooling [14] 88.35 ± 0.26 57.28 ± 0.31 Concatenation [27] 83.40 ± 0.48 52.91 ± 0.59 Self-Attention [32] 79.65 ± 0.68 48.60 ± 0.70 CSN [28] 84.90 ± 0.52 57.06 ± 1.70 BiLSTM [5] 74.44 ± 0.95 45.41 ± 0.40 BiLSTM+VSE [5] 74.82 ± 0.63 46.02 ± 0.62 Ours 91.90 ± 0.40 64.35 ± 0.92 Table 2: Outfit compatibility prediction AUC and FITB accuracy on Polyvore-T dataset.…”

“…Many works used multimedia information to answer the questions like "Is this outfit compatible?" [14,27,28], "How to generate a compatible outfit?" [5,19].…”

“…To effectively implement the comparison module, we follow the ideas from the projected embedding [3] and type-aware embedding [28]. To leverage both image and text information of fashion items, we adopt visual semantic embedding [12] to learn a common representation between them.…”