Towards Intelligent Retail: Automated on-Shelf Availability Estimation Using a Depth Camera

Abstract

Efficient management of on-shelf availability and inventory is a key issue to achieve customer satisfaction and reduce the risk of profit loss for both retailers and manufacturers. Conventional store audits based on physical inspection of shelves are labor-intensive and do not provide reliable assessment. This paper describes a novel framework for automated shelf monitoring, using a consumer-grade depth sensor. The aim is to develop a low-cost embedded system for early detection of out-of-stock situations with particular regard to perishable goods stored in countertop shelves, refrigerated counters, baskets or crates. The proposed solution exploits 3D point cloud reconstruction and modelling techniques, including surface fitting and occupancy grids, to estimate product availability, based on the comparison between a reference model of the shelf and its current status. No a priori knowledge about the product type is required, while the shelf reference model is automatically learnt based on an initial training stage. The output of the system can be used to generate alerts for store managers, as well as to continuously update product availability estimates for automated stock ordering and replenishment and for e-commerce apps. Experimental tests performed in a real retail environment show that the proposed system is able to estimate the on-shelf availability percentage of different fresh products with a maximum average discrepancy with respect to the actual one of about 5.0%. INDEX TERMS Automated stock monitoring, intelligent retail, RGB-D sensors, 3D reconstruction and modeling.

Enhanced Out-of-Stock Detection in Retail Shelf Images Based on Deep Learning

“…Milella et al. [ 7 ] presented a solution for OSA estimation using 3D data provided by a depth sensor. Moorthy et al.…”

“…If many items of a unique product have been sold since opening or, even more precisely, if the ratio of sold items to displayed items of a product is large, store employees should be notified to replenish the appropriate shelf. Alternatively, different sensors can be installed and utilized to detect changes on store shelves and possible OOSs, such as radio frequency identification (RFID) technology [ 4 ], infrared (IR) sensors [ 6 ], or depth sensors [ 7 ]. Furthermore, computer vision methods can be utilized to detect OOS in images of store shelves.…”

Designing an Efficient End-to-end Machine Learning Pipeline for Real-time Empty-shelf Detection

“…Rosado et al [12] proposed a supervised machine learning method leveraging Support Vector Machines for OOS detection by leveraging the geometrical and visual features in a high-resolution panoramic shelf images of grocery retail stores. Milella et al [14] developed an early detection method for OOS situations based on low-cost embedded system by exploiting 3D point cloud reconstruction and modeling techniques. Pranwada et al [15] presented a computer vision and machine learning based approach for detecting empty shelves from camera images.…”

“…Such methods range from manual store audits, leveraging of Radio Frequency Identification (RFID) and RFID reader integrated weight sensing mat to ZigBee transceiver and consumer-grade depth sensor to customer-centric approach of scanning the QR code and alerting the store manager [8][9][10]; such methods are either manual or not cost-effective to integrate into existing systems [11]. There have also been proposal to address the OOS issue using image processing and traditional machine learning algorithms such as using image processing [10], supervised learning using Support Vector Machines (SVM) [12], blob detection followed by discriminative machine learning [13], 3D point cloud reconstruction and modeling [14], and computer vision [15]. Nevertheless, such traditional ML approaches have low accuracy even using larger datasets and are difficult to make better.…”

Computer Vision Based Planogram Compliance Evaluation

“…Due to manual planogram compliance evaluation being both laborious and error-prone, various approaches have been attempted to automate the task. These include inventorybased methods, tagging products with RFID tags (Saran et al, 2015) and utilizing specialist equipment such as depth cameras (Milella et al, 2020). Many of these methods have been dismissed as either not accurate enough or as too costly (Marder et al, 2015).…”