Product Recognition in Store Shelves as a Sub-Graph Isomorphism Problem

Abstract

The arrangement of products in store shelves is carefully planned to maximize sales and keep customers happy. However, verifying compliance of real shelves to the ideal layout is a costly task routinely performed by the store personnel. In this paper, we propose a computer vision pipeline to recognize products on shelves and verify compliance to the planned layout. We deploy local invariant features together with a novel formulation of the product recognition problem as a sub-graph isomorphism between the items appearing in the given image and the ideal layout. This allows for auto-localizing the given image within the aisle or store and improving recognition dramatically. 1 missing or misplaced items. Key to our approach is a novel formulation of the problem as a sub-graph isomorphism between the product detected in the given image and those that should ideally be found therein given the planogram. Accordingly, our pipeline relies on a standard feature-based object recognition step, followed by the novel graph-based consistency check and a final localized image search to improve the overall product recognition rate.

Domain invariant hierarchical embedding for grocery products recognition

“…However, their experiments report performance far from conducive to real-world deployment. A number of more recent proposals are aimed at improving product recognition by leveraging on: a) stronger features followed by classification ( [3]), b) the statistical correlation between nearby products on shelves ( [1,2]) c) information on the expected product disposition ( [33]) or d) a hierarchical multi-stage recognition pipeline ( [4]). Yet, all these recent works focus on a relatively small-scale problem, i.e.…”

“…As already discussed, at test time we pursue recognition from a different set of images (query). To create this set, for Grocery Products we automatically cropped individual items from the available shelf images according to the annotation released in [33], thereby obtaining a total of 938 query images. As for Prod-uct8600, we cropped and annotated individual items from shelf videos that we acquired in a grocery store by a tablet camera, for a total number of 273 query images.…”

A deep learning pipeline for product recognition on store shelves

“…However, their experiments report performance far from conducive to real-world deployment in terms of accuracy and speed. A number of more recent works tried to improve product recognition by leveraging on: a) stronger features followed by classification [4], b) the statistical correlation between nearby products on shelves [1,2] c) additional information on the expected product disposition [22] or d) a hierarchical multi-stage recognition pipeline [5]. Yet, all these recent papers focus on a relatively small-scale problem, i.e.…”

“…In particular, given the candidate regions extracted from the query image and their corresponding sets of K-NN, we consider the 1-NN of the region proposals extracted with a high confidence (> 0.1) by the Detector in order to find the main macro category of the image. Then, in case the (a)-Customer [6] (b)-Management [22] majority of detections votes for the same macro category, it is safe to assume that the pictured shelf contains almost exclusively items of that category thus filter the K-NN for all candidate regions accordingly. It is worth observing how this strategy implicitly leverages on those products easier to identify (i.e., the high-confidence detections) to increase the chance to correctly recognize the harder ones.…”

“…To compare our pipeline with respect to FS, we use the annotations provided by the authors and perform recognition against the smaller reference database of 182 products used in [22]. The results are reported in Table 2.…”