Collaborative Image Understanding

Abstract

Automatically understanding the contents of an image is a highly relevant problem in practice. In e-commerce and social media settings, for example, a common problem is to automatically categorize user-provided pictures. Nowadays, a standard approach is to fine-tune pre-trained image models with application-specific data. Besides images, organizations however often also collect collaborative signals in the context of their application, in particular how users interacted with the provided online content, e.g., in forms of viewing, rating, or tagging. Such signals are commonly used for item recommendation, typically by deriving latent user and item representations from the data. In this work, we show that such collaborative information can be leveraged to improve the classification process of new images. Specifically, we propose a multitask learning framework, where the auxiliary task is to reconstruct collaborative latent item representations. A series of experiments on datasets from e-commerce and social media demonstrates that considering collaborative signals helps to significantly improve the performance of the main task of image classification by up to 9.1%.

Semi-supervised Adversarial Learning for Complementary Item Recommendation

“…However, we assume that each item has a known category and there is some additional side-information available for each item. Moreover, we make the assumption that there exists a labeled set of pairs of complementary items 4 . Such a set may be created manually or automatically derived, e.g., by analyzing co-purchase patterns of pairs of items with sufficient purchase signals.…”

“…In future work, we plan to extend our model to also support warm items, using the same framework. That way, the input features for the item encoder would include also collaborative data by applying e.g., the method suggested in [4]. In addition to such extensions and continuing related research in [39,46], other promising approaches to further improve the effectiveness of the model could lie in the personalization of the complementary item recommendations and to consider aspects of item quality.…”

“…In practice, there can be multiple complementary categories and the recommendation process may thus be repeated for each category 4. Note that the pairs in the labeled set do not contain the seed item.…”