Mobile Visual Assistive Apps: Benchmarks of Vision Algorithm Performance

Abstract

Although the use of computer vision to analyse images from smartphones is in its infancy, the opportunity to exploit these devices for various assistive applications is beginning to emerge. In this paper, we consider two potential applications of computer vision in the assistive context for blind and partially sighted users. These two applications are intended to help provide answers to the questions of "Where am I?" and "What am I holding?". First, we suggest how to go about providing estimates of the indoor location of a user through queries submitted by a smartphone camera against a database of visual paths-descriptions of the visual appearance of common journeys that might be taken. Our proposal is that such journeys could be harvested from, for example, sighted volunteers. Initial tests using bootstrap statistics do indeed suggest that there is sufficient information within such visual path data to provide indications of: a) along which of several routes a user might be navigating; b) where along a particular path they might be. We will also discuss a pilot benchmarking database and test set for answering the second question of "What am I holding?". We evaluated the role of video sequences, rather than individual images, in such a query context, and suggest how the extra information provided by temporal structure could significantly improve the reliability of search results, an important consideration for assistive applications.

The Last 10 Metres: Using Visual Analysis and Verbal Communication in Guiding Visually Impaired Smartphone Users to Entrances

“…In a pilot study (Rivera-Rubio et al, 2013), conducted within indoor, but highly ambiguous corridors, we have found that with relatively modest processes, paths can be distinguished with reasonable certainty using visual cues alone. In more extensive tests, verified with surveying equipment (Rivera-Rubio et al, 2014), we found that user location on a path can be inferred from hand-held and wearable cameras.…”

“…Locating user on map based on visual cues: The task is to locate the user precisely on the map (within a given radius determined on the basis of GPS output) by identifying landmarks and visual cues in usergenerated live feed and matching these to the tags and images in the semantically enriched local maps. In a pilot study (Rivera-Rubio et al, 2013), conducted within indoor, but highly ambiguous corridors, we have found that with relatively modest processes, paths can be distinguished with reasonable certainty using visual cues alone. In more extensive tests, verified with surveying equipment (Rivera-Rubio et al, 2014), we found that user location on a path can be inferred from hand-held and wearable cameras.…”

Appearance-based indoor localization: A comparison of patch descriptor performance

“…along a specific physical path, relative to start and end point, a person might be. We addressed the first of these questions in previous work (Rivera-Rubio et al, 2013).…”

Fundamental controls on fluid flow in carbonates: current workflows to emerging technologies

“…Machine learning and pattern recognition are not new to the geosciences but advances in AI (and related advances in robotics) have a significant potential to impact future workflows. Systems that are being developed to assist the blind through computer visualization and sensor data fusion (Rivera-Rubio et al 2013), security monitoring systems (Choi & Savarese 2014) and biomedical imaging (Mudry et al 2013;Toews & Fig. 11.…”