Who, Where, and What to Wear?

Abstract

Fashion knowledge helps people to dress properly and addresses not only physiological needs of users, but also the demands of social activities and conventions. It usually involves three mutually related aspects of: occasion, person and clothing. However, there are few works focusing on extracting such knowledge, which will greatly benefit many downstream applications, such as fashion recommendation. In this paper, we propose a novel method to automatically harvest fashion knowledge from social media. We unify three tasks of occasion, person and clothing discovery from multiple modalities of images, texts and metadata. For person detection and analysis, we use the off-the-shelf tools due to their flexibility and satisfactory performance. For clothing recognition and occasion prediction, we unify the two tasks by using a contextualized fashion concept learning module, which captures the dependencies and correlations among different fashion concepts. To alleviate the heavy burden of human annotations, we introduce a weak label modeling module which can effectively exploit machine-labeled data, a complementary of clean data. In experiments, we contribute a benchmark dataset and conduct extensive experiments from both quantitative and qualitative perspectives. The results demonstrate the effectiveness of our model in fashion concept prediction, and the usefulness of extracted knowledge with comprehensive analysis. CCS CONCEPTS• Information systems → Specialized information retrieval.

Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification

“…Then, we explore how to leverage the retrieved comments in multimodal classification and exploit a self-training framework to identify comments' hints which shape the cross-modal understanding (henceforth comment-aware self-training). This considers method feasibility in scenarios where large-scale labeled data is unavailable, which commonly appears in the realistic practice, because the annotation for multimodal data from social media is extremely expensive (Ma et al, 2019). Concretely, we adopt a teacher-student prototype (Meng et al, 2020;Shen et al, 2021) and tailor-make it to learn multimodal understanding with the help of user comments.…”

“…The labeled dataset L is usually limited in scales (Ma et al, 2019), posing the over-fitting concern. Meanwhile the retrieved posts, similar to the data in L, could form an unlabeled set (U = {x ′ i , c i } Kl i=1 ) to enrich training data.…”

Fashion Recommendation Systems, Models and Methods: A Review

“…According to different studies, e-commerce retailers, such as Amazon, eBay, and Shopstyle, and social networking sites, such as Pinterest, Snapchat, Instagram, Facebook, Chictopia, and Lookbook, are now regarded as the most popular media for fashion advice and recommendations [15][16][17][18][19][20][21][22]. Research on textual content, such as posts and comments [23], emotion and information diffusion [24], and images has attracted the attention of modernday researchers, as it can help to predict fashion trends and facilitate the development of effective recommendation systems [5,[25][26][27]]. An effective recommendation system is a crucial tool for successfully conducting an e-commerce business.…”