Learning visual similarity for product design with convolutional neural networks

Abstract

a) Query 1: Input scene and box (b) Project into 256D embedding (c) Results 2: use of product in-situ Convolutional Neural Network Learned Parameters θ (a) Query 2: Product (c) Results 1: visually similar products Figure 1: Visual search using a learned embedding. Query 1: given an input box in a photo (a), we crop and project into an embedding (b) using a trained convolutional neural network (CNN) and return the most visually similar products (c). Query 2: we apply the same method to search for in-situ examples of a product in designer photographs. The CNN is trained from pairs of internet images, and the boxes are collected using crowdsourcing. The 256D embedding is visualized in 2D with t-SNE. Photo credit: Crisp Architects and Rob Karosis (photographer). AbstractPopular sites like Houzz, Pinterest, and LikeThatDecor, have communities of users helping each other answer questions about products in images. In this paper we learn an embedding for visual search in interior design. Our embedding contains two different domains of product images: products cropped from internet scenes, and products in their iconic form. With such a multi-domain embedding, we demonstrate several applications of visual search including identifying products in scenes and finding stylistically similar products. To obtain the embedding, we train a convolutional neural network on pairs of images. We explore several training architectures including re-purposing object classifiers, using siamese networks, and using multitask learning. We evaluate our search quantitatively and qualitatively and demonstrate high quality results for search across multiple visual domains, enabling new applications in interior design.

SIFT Meets CNN: A Decade Survey of Instance Retrieval

“…[137] 112,623 n.a Landmark MV RGB-D [142] 250,000 300 House. object Product [143] 101,945×2 n.a Furniture …”

“…Another dataset called Tokyo Time Machine is collected using Google Street View Time Machine which provides images depicting the same places over time [137]. While most of the above datasets focus on landmarks, Bell et al [143] build a Product dataset consisting of furniture by developing a crowd-sourced pipeline to draw connections between in-situ objects and the corresponding products. It is also feasible to fine-tune on the query sets suggested in [144], but this method may not be adaptable to new query types.…”

“…A standard siamese network based on AlexNet and the contrastive loss is employed in [143]. In [24], Radenović et al propose to replace the FC layers with a MAC layer [10].…”

Regularized deep learning for fashion and social media data analysis

“…Deep feature embedding with convolutional neural networks attracts a lot of attention recently, especially contrastive embedding [57] and triplet embedding [58]. Bell et al…”

“…For example, Kiapour et al applied CNN features as image representation, and calculated the cross-entropy loss measuring whether two images are matched or non-matched. Siamese neural networks have attracted a lot of attentions recently, such as contrastive embedding [57] and triplet embedding [58].…”