A deep learning pipeline for product recognition on store shelves

Tonioni, Alessio; Serra, Eugenio; Stefano, Luigi Di

doi:10.1109/ipas.2018.8708890

Cited by 39 publications

(53 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the retail domain, several prior works have tackled the item identification problem (Qiao et al, 2017;Tonioni et al, 2018;Klasson et al, 2019;Wei et al, 2019) but current stateof-the-art is still limited in accuracy due to the great similarity in appearance across products (e.g., chocolate bars of the same brand with different sizes, flavors or textures). On top of that, none of these works take videos as input, meaning an added layer of scene understanding is required in order to figure out which of the detected products is the one the customer took (vs. products which remain on the shelf).…”

Section: Vision-based Object Identificationmentioning

confidence: 99%

FAIM: Vision and Weight Sensing Fusion Framework for Autonomous Inventory Monitoring in Convenience Stores

Falcão

Ruiz²,

Pan

et al. 2020

Front. Built Environ.

View full text Add to dashboard Cite

A common pain point for physical retail stores is live inventory monitoring, i.e., knowing how many items of each product are left on the shelves. About 4% of sales are lost due to an average 5-10% out-of-shelf stockout rate, while additional supplies existed in the warehouse. Traditional techniques rely on manual inspection, per-item tagging using RFIDs, or human-in-the-loop systems, such as Amazon Go. These approaches, while effective, either have poor accuracy, long delays between results or are cost prohibitive. In this paper, we present FAIM (Autonomous Inventory Monitoring Framework) for cashier-less stores. To the best of our knowledge, this is the first fully autonomous system that fuses multiple sensing modalities. Utilizing weight difference on a shelf, visual item recognition in customers' hands and prior knowledge of item layout FAIM monitors products picked up or returned without human-in-the-loop. We present results from a real-world setup with 85 items (33 unique products) replicating the layout of a local 7-Eleven store. To evaluate our system we characterize the similarity of the unique products across three physical features (i.e., weight, color, and location). Our results show that the fused approach provides up to 92.6% item identification accuracy, a 2× reduction in error compared to reported self-checkout stations.

show abstract

Section: Vision-based Object Identificationmentioning

confidence: 99%

FAIM: Vision and Weight Sensing Fusion Framework for Autonomous Inventory Monitoring in Convenience Stores

Falcão

Ruiz²,

Pan

et al. 2020

Front. Built Environ.

View full text Add to dashboard Cite

show abstract

“…Recently, recognition of products on retail shelves has become an interesting research topic in computer vision [4]- [6], [12]- [21]. Several commercial product search systems exist and obtain good classification results on some product categories with specific planar shapes and textures such as CDs and books [12], [13].…”

Section: A Retail Product Recognitionmentioning

confidence: 99%

“…Several commercial product search systems exist and obtain good classification results on some product categories with specific planar shapes and textures such as CDs and books [12], [13]. The methods in [4]- [6], [14]- [16], [18]- [21] focus on retail product recognition on shelves.…”

Section: A Retail Product Recognitionmentioning

confidence: 99%

Context-Aware Confidence Sets for Fine-Grained Product Recognition

2019

View full text Add to dashboard Cite

We present a new approach for fine-grained classification of retail products, which learns and exploits statistical context information about likely product arrangements on shelves, incorporates visual hierarchies across brands, and returns recognition results as ''confidence sets'' that are guaranteed to contain the true class at a given confidence level. Our system consists of three important components: 1) a nested hierarchy of product classes are automatically constructed based on visual similarities, 2) a confidence set predictor is trained based on class posteriors by using coarse-to-fine binary classifiers to discriminate each nested cluster of the hierarchy from the remainder of classes and a Bayesian network (BN) model that encodes the joint distribution of classifier scores with the fine-level class variable, and 3) n hidden Markov model (HMM) is trained with nested hidden states from the class hierarchy to model spatial transition across the nodes of product class hierarchy and resolve errors in the context-free confidence set results. Novel aspects of the proposed method include 1) combining confidence sets and context information via a HMM, 2) applying this concept to fine grained recognition of products arranged in retail shelves, and 3) presenting experimental results on four large datasets, collected from actual retail stores. We compare our approach with existing confidence set approaches and state-of-the-art convolutional neural networks classifiers including SENet-154, DenseNet-161, B-CNN, and Inception-Resnet-v2. Our approach performs comparably or better than state-of-the-art deep classifiers and exhibits high accuracy for relatively small confidence set sizes.

show abstract

“…Therefore, research on identification of packaged products relies on relatively few, rather small and quite old datasets [9,17]. Existing studies on identifying packaged products via computer vision indicate promising potential [8,17,43,44], but they rely on such limited datasets and are conducted under resource-intense lab conditions, and do therefore not prove real-world applicability of computer vision based product identification. Although standards on product identifiers (e.g.…”

Section: Motivationmentioning

confidence: 99%

Towards Identification of Packaged Products via Computer Vision

Fuchs

Grundmann

Fleisch

2019

Proceedings of the 9th International Conference on the Internet of Things

View full text Add to dashboard Cite

Identification of packaged products in retail environments still relies on barcodes, requiring active user input and limited to one product at a time. Computer vision (CV) has already enabled many applications, but has so far been under-discussed in the retail domain, albeit allowing for faster, hands-free, more natural humanobject interaction (e.g. via mixed reality headsets). To assess the potential of current convolutional neural network (CNN) architectures to reliably identify packaged products within a retail environment, we created and open-source a dataset of 300 images of vending machines with 15k labeled instances of 90 products. We assessed observed accuracies from transfer learning for imagebased product classification (IC) and multi-product object detection (OD) on multiple CNN architectures, and the number of images instances required per product to achieve meaningful predictions. Results show that as little as six images are enough for 90% IC accuracy, but around 30 images are needed for 95% IC accuracy. For simultaneous OD, 42 instances per product are necessary and far more than 100 instances to produce robust results. Thus, this study demonstrates that even in realistic, fast-paced retail environments, image-based product identification provides an alternative to barcodes, especially for use-cases that do not require perfect 100% accuracy.

show abstract

A deep learning pipeline for product recognition on store shelves

Cited by 39 publications

References 21 publications

FAIM: Vision and Weight Sensing Fusion Framework for Autonomous Inventory Monitoring in Convenience Stores

FAIM: Vision and Weight Sensing Fusion Framework for Autonomous Inventory Monitoring in Convenience Stores

Context-Aware Confidence Sets for Fine-Grained Product Recognition

Towards Identification of Packaged Products via Computer Vision

Contact Info

Product

Resources

About