Mining Fashion Outfit Composition Using an End-to-End Deep Learning Approach on Set Data

Abstract

Composing fashion outfits involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and meta-data. We propose to leverage outfit popularity on fashion oriented websites to supervise the scoring component. The scoring component is a multi-modal multi-instance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.

POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion

“…Methods belonging to the second category, such as presented in [11] and [3], are based on modeling a fashion out t as a set or an ordered sequence. Li et al deploy an end-to-end deep learning system which can classify a given out t as popular or unpopular [11]. Han et al train a bidirectional LSTM model to sequentially generate out ts [3].…”

“…e concept of fashion relies mostly on visual and textual information. Most previous works suggest to leverage image and text to learn multi-modal embeddings [3,11]. In our work, we use We mask the items in the out t one at a time.…”

Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network

“…For end-to-end training outfits, pooling [14] or concatenation [27] operations can be used to aggregate multiple item features. Then the multilayer perceptron (MLP) can be used to compute a compatibility score.…”

“…Pooling [14] 88.35 ± 0.26 57.28 ± 0.31 Concatenation [27] 83.40 ± 0.48 52.91 ± 0.59 Self-Attention [32] 79.65 ± 0.68 48.60 ± 0.70 CSN [28] 84.90 ± 0.52 57.06 ± 1.70 BiLSTM [5] 74.44 ± 0.95 45.41 ± 0.40 BiLSTM+VSE [5] 74.82 ± 0.63 46.02 ± 0.62 Ours 91.90 ± 0.40 64.35 ± 0.92 Table 2: Outfit compatibility prediction AUC and FITB accuracy on Polyvore-T dataset.…”

“…We compare our method with several recent fashion outfit compatibility works. Their brief descriptions are as follows: Pooling [14]: It uses average-pooling operation to aggregate the variable length of feature inputs to predict the compatibility. Concatenation [27]: It concatenates 5 item features into a long vector with a length of 1000 × 5, then uses MLP as the binary classifier.…”