Domain invariant hierarchical embedding for grocery products recognition

Abstract

Recognizing packaged grocery products based solely on appearance is still an open issue for modern computer vision systems due to peculiar challenges. Firstly, the number of different items to be recognized is huge (i.e., in the order of thousands) and rapidly changing over time. Moreover, there exist a significant domain shift between the images that should be recognized at test time, taken in stores by cheap cameras, and those available for training, usually just one or a few studio-quality images per product. We propose an end-to-end architecture comprising a GAN to address the domain shift at training time and a deep CNN trained on the samples generated by the GAN to learn an embedding of product images that enforces a hierarchy between product categories. At test time, we perform recognition by means of K-NN search against a database consisting of just one reference image per product. Experiments addressing recognition of products present in the training datasets as well as different ones unseen at training time show that our approach compares favourably to state-ofthe-art methods on the grocery recognition task and generalize fairly well to similar ones.

Deep Learning for Retail Product Recognition: Challenges and Techniques

“…Although this method does not aim to predict the product categories, it can accurately perform the object proposal detection for the products with different scale ratios in one image, which is a practical issue in supermarket scenarios. In [ 71 ], authors considered three different popular CNN models, VGG-16 [ 42 ], ResNet [ 43 ], and Inception [ 70 ], in their approach and performed the K-NN similarity search extensively with the output of the three models. Their method was evaluated with three grocery product datasets, and the largest one contained 938 classes of food items.…”

“…Although GANs have shown compelling results in the domains of general object classification and detection, there are very few works using GANs for product recognition. To the best of our knowledge, there are only three papers [ 7 , 71 , 78 ] attempting to exploit GANs to create new images in the field of product recognition. In the work of [ 7 ], the authors proposed a large-scale checkout dataset containing synthetic training images generated by CycleGAN [ 106 ].…”

Towards Identification of Packaged Products via Computer Vision

“…Therefore, research on identification of packaged products relies on relatively few, rather small and quite old datasets [9,17]. Existing studies on identifying packaged products via computer vision indicate promising potential [8,17,43,44], but they rely on such limited datasets and are conducted under resource-intense lab conditions, and do therefore not prove real-world applicability of computer vision based product identification. Although standards on product identifiers (e.g.…”

Impact of Mixed Reality Food Labels on Product Selection

“…MR headsets such as Microsoft HoloLens or MagicLeap One are expected to be increasingly adopted by consumers in the near future and offer multiple advantage in supporting healthy food choices. For example, MR headsets feature multiple continuously active cameras that scan their nearby environment and can discover packaged products through computer vision without requiring active user input [40]. Such MR headsets also allow for the display of three-dimensional visualization of interventions, because spatial computing allows for relative positioning of visualizations to the user's field of view and detected objects, thereby achieving high degrees of presence, salience and immersion, important prerequisites for interventions.…”

“…MobileNet [34], ResNet [17], DenseNet [18]) or more recently, generative adversarial networks (GAN) [3]. Different, recent studies have demonstrated the feasibility to recognize packaged products through computer vision on a large scale from labelled images [14,26,39,40].…”