Learning a Unified Embedding for Visual Search at Pinterest

Abstract

At Pinterest, we utilize image embeddings throughout our search and recommendation systems to help our users navigate through visual content by powering experiences like browsing of related content and searching for exact products for shopping. In this work we describe a multi-task deep metric learning system to learn a single unified image embedding which can be used to power our multiple visual search products. The solution we present not only allows us to train for multiple application objectives in a single deep neural network architecture, but takes advantage of correlated information in the combination of all training data from each application to generate a unified embedding that outperforms all specialized embeddings previously deployed for each product. We discuss the challenges of handling images from different domains such as camera photos, high quality web images, and clean product catalog images. We also detail how to jointly train for multiple product objectives and how to leverage both engagement data and human labeled data. In addition, our trained embeddings can also be binarized for efficient storage and retrieval without compromising precision and recall. Through comprehensive evaluations on offline metrics, user studies, and online A/B experiments, we demonstrate that our proposed unified embedding improves both relevance and engagement of our visual search products for both browsing and searching purposes when compared to existing specialized embeddings. Finally, the deployment of the unified embedding at Pinterest has drastically reduced the operation and engineering cost of maintaining multiple embeddings while improving quality.

Bootstrapping Complete The Look at Pinterest

“…The visual featurizer uses the same SE-ResNext101 backbone [14] as Pinterest's Unified Embedding [37]. Unified Embedding is a multi-task learning model that has been trained for three visual discovery tasks: Flashlight [2], Lens [36], and Shop The Look [7].…”

“…The motivation for the classification network is its similarity to the Unified Embedding model [37], if for example, we decide to add CTL as an additional task to our Unified Embedding training. In the classification model, each outfit has a unique instance label.…”

“…This is done to dissuade learning visual similarity in place of style compatibility. We then optimize using contrastive loss: Figure 8: We take this figure directly from the Unified Embedding paper [37]. It depicts how we get a subset of "proxy" instances for calculating classification loss.…”

“…In recent years, visual search products such as Shop The Look [7] helped to close this loop by enabling users to make purchases directly from pins they have saved. Shop The Look uses the Unified Embedding model [37] trained on millions of pieces of Pinterest content to find products that are visually similar to the objects detected in the image.…”

Zero-Shot Heterogeneous Transfer Learning from Recommender Systems to Cold-Start Search Retrieval

“…Recently neural information retrieval (neural IR) models have been widely applied in search products across many industries [10,22,32]. Such methods can retrieve and score items that do not share keywords with the query.…”