Learning Visual Clothing Style with Heterogeneous Dyadic Co-Occurrences

Abstract

With the rapid proliferation of smart mobile devices, users now take millions of photos every day. These include large numbers of clothing and accessory images. We would like to answer questions like 'What outfit goes well with this pair of shoes?' To answer these types of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer these types of questions. The main idea of this framework is to learn a feature transformation from images of items into a latent space that expresses compatibility. For the feature transformation, we use a Siamese Convolutional Neural Network (CNN) architecture, where training examples are pairs of items that are either compatible or incompatible. We model compatibility based on co-occurrence in largescale user behavior data; in particular co-purchase data from Amazon.com. To learn cross-category fit, we introduce a strategic method to sample training data, where pairs of items are heterogeneous dyads, i.e., the two elements of a pair belong to different high-level categories. While this approach is applicable to a wide variety of settings, we focus on the representative problem of learning compatible clothing style. Our results indicate that the proposed framework is capable of learning semantic information about visual style and is able to generate outfits of clothes, with items from different categories, that go well together.

A Large-Scale Deep Architecture for Personalized Grocery Basket Recommendations

“…There has also been significant research to infer functionally complementary relations for item-item recommendation tasks. These models focus on learning compatibility [11], complementarity [12,13,14], and complementary-similarity [15,16] relations…”

Diversity in Fashion Recommendation Using Semantic Parsing

“…finding similarity between two images using Siamese network [1,2,3]. In an extension of the work, it is also possible to learn similarity-dissimilarity between three images using a Triplet network [4,5].…”

Regularized deep learning for fashion and social media data analysis

“…Although there are some related works of accessories recommendation [94,95,103], we are not with the same scenario and goal as them. [94,95] focus on predicting what else will the user buy according to his/her purchase histories. Since they apply co-purchase data as the source, all the recommended results may with the same category or style as the query image, meaning less diversity.…”

“…3.1 (b), the second task is which accessories should he or she buy to pair this item? Existing accessories recommendation in online shopping mainly used co-purchase data to learn pairing models [94,95]. However, actually, in daily life, people may not always wear co-purchased clothing together.…”

“…So far, few works have focused on the problem of accessories recommendation in online shopping scenario [94,95]. McAuley et al collected a large-scale co-purchase dataset from Amazon.com [95].…”