ShopTalk: Independent Blind Shopping Through Verbal Route Directions and Barcode Scans

Abstract

Independent shopping in modern grocery stores that carry thousands of products is a great challenge for people with visual impairments. ShopTalk is a proof-of-concept wearable system designed to assist visually impaired shoppers with finding shelved products in grocery stores. Using synthetic verbal route directions and descriptions of the store layout, ShopTalk leverages the everyday orientation and mobility skills of independent visually impaired travelers to direct them to aisles with target products. Inside aisles, an off-the-shelf barcode scanner is used in conjunction with a software data structure, called a barcode connectivity matrix, to locate target product on shelves. Two experiments were performed at a real world supermarket. A successful earlier single-subject experiment is summarized and a new experiment involving ten visually impaired participants is presented. In both experiments, ShopTalk was successfully used to guide visually impaired shoppers to multiple products located in aisles on shelves. ShopTalk is a feasible system for guiding visually impaired shoppers who are skilled, independent travelers. Its design does not require any hardware instrumentation of the store and leads to low installation and maintenance costs.

Accessible Shopping Systems for Blind and Visually Impaired Individuals: Design Requirements and the State of the Art

“…The product retrieval rate was 100%. All ten participants found all three products in every run [2].…”

“…ShopTalk was based on the following conceptual formula: independent blind shopping = verbal route instructions + shelf barcode scans [16]. The system was our first attempt (and, to the best of our knowledge, the first attempt reported in the accessible shopping literature) to use MSI (Modified Plessey) shelf barcodes (See Fig.…”

“…Some staffers are unfamiliar with the store layout, others become irritated with long searches and requests to read aloud product ingredients, and still others do not have adequate English skills to read the products' ingredients or answer basic questions about the supermarket layout. These difficulties cause blind and VI shoppers to abandon searching for desirable products, settle for distant substitutes, or, in the worst case, abandon independent shopping altogether [2].…”

A simple and efficient approach to barcode localization

“…That is, these inventory systems place barcodes on the shelves immediately beneath every product area. Nicholson and Kulyukin [1] presented ShopTalk, a wearable small-scale system as shown in Figure 1 that enables a visually impaired shopper to successfully retrieve common grocery products using verbal route direction and barcode scans. In ShopTalk, the visually impaired shopper will use the off-theshelf laser based barcode reader to read the barcodes from the shelves and transmit data to a computing device for estimating his location within the store.…”

User Intent Communication in Robot-Assisted Shopping for the Blind

“…Although the accessibility of cell phones for people with visual impairments remains an issue, the situation has been improving as more and more individuals with visual impairments become cell phone users. We hope that in the future visually impaired shoppers will communicate with RobotCart using their cell phones (Nicholson et al, 2009;Nicholson & Kulyukin, 2007). The remainder of the chapter is organized as follows.…”