Improving Outfit Recommendation with Co-supervision of Fashion Generation

Abstract

The task of fashion recommendation includes two main challenges: visual understanding and visual matching. Visual understanding aims to extract effective visual features. Visual matching aims to model a human notion of compatibility to compute a match between fashion items. Most previous studies rely on recommendation loss alone to guide visual understanding and matching. Although the features captured by these methods describe basic characteristics (e.g., color, texture, shape) of the input items, they are not directly related to the visual signals of the output items (to be recommended). This is problematic because the aesthetic characteristics (e.g., style, design), based on which we can directly infer the output items, are lacking. Features are learned under the recommendation loss alone, where the supervision signal is simply whether the given two items are matched or not.To address this problem, we propose a neural co-supervision learning framework, called the FAshion Recommendation Machine (FARM). FARM improves visual understanding by incorporating the supervision of generation loss, which we hypothesize to be able to better encode aesthetic information. FARM enhances visual matching by introducing a novel layer-to-layer matching mechanism to fuse aesthetic information more effectively, and meanwhile avoiding paying too much attention to the generation quality and ignoring the recommendation performance.Extensive experiments on two publicly available datasets show that FARM outperforms state-of-the-art models on outfit recommendation, in terms of AUC and MRR. Detailed analyses of generated and recommended items demonstrate that FARM can encode better features and generate high quality images as references to improve recommendation performance.

Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network for Personalized Image Caption

“…(Zeng et al 2019) utilizes user descriptions such as age and gender to generate social media comments. In addition to the above text generation which commonly has a similar encoder-decoder framework (Sutskever, Vinyals, and Le 2014), (Lin et al 2019) investigate the personalized fashion generation by generating images through a deconvolutional neural network (Zeiler, Taylor, and Fergus 2011). (Wang, Zhang, and He 2019) synthesizes continuous states and medication dosages of patients with generative adversarial networks.…”

Learning Context-Aware Outfit Recommendation

“…If the style is identified by low-dimensional visual similarity [2,9], it may be too specific to detect similar-style images, and the hand-made style may be too abstract to capture subtle style differences. The clothing style is different from clothing shape (sweater, dress) or clothing attribute [10][11][12]. In the study of clothing style recognition, the early method [13,14] explored supervised learning and classify the style according to the user's identity information [15].…”

Auxiliary Template-Enhanced Generative Compatibility Modeling

“…For example, the "Chiffon Blouse with Bow Detail" matches well with the "High-waisted Pleated Design Midi Skirt", while the "Striped Shirt" goes well with the "Fray Hem Denim Wide Leg Pants". To well capture the distinguished features of items, we adopt the global average pooling (GAP) [Lin et al, 2013] for its powerful capability in locating the discriminant areas of an image. According to GAP, each feature map with the shape of w × h would be averaged to one value.…”