Real time object scanning using a mobile phone and cloud-based visual search engine

Abstract

Computer vision and human-powered services can provide blind people access to visual information in the world around them, but their efficacy is dependent on high-quality photo inputs. Blind people often have difficulty capturing the information necessary for these applications to work because they cannot see what they are taking a picture of. In this paper, we present Scan Search, a mobile application that offers a new way for blind people to take high-quality photos to support recognition tasks. To support realtime scanning of objects, we developed a key frame extraction algorithm that automatically retrieves high-quality frames from continuous camera video stream of mobile phones. Those key frames are streamed to a cloud-based recognition engine that identifies the most significant object inside the picture. This way, blind users can scan for objects of interest and hear potential results in real time. We also present a study exploring the tradeoffs in how many photos are sent, and conduct a user study with 8 blind participants that compares Scan Search with a standard photo-snapping interface. Our results show that Scan Search allows users to capture objects of interest more efficiently and is preferred by users to the standard interface.

RegionSpeak

“…As seen in both related work [24] and our experiments, blind people usually have worse phototography skills than sighted people, so an accessible camera interface with few restric tions and rich guidance is crucial to help them take better photos. Existing camera interfaces often fail to provide assis tance, resulting in poor performance of photo-based assistive applications.…”

“…al's key frame extraction algorithm [24], we created an panorama interface for RegionSpeak which has no restriction that needs visual inspection. Users of RegionSpeak can move the camera in any direction, and the key frame extraction al gorithm will detect substantial changes in view port and alert users to hold their position to capture a new image.…”

Privacy-preserving Twitter-based Solution for Visually Impaired People

“…These include ThirdEye [5], VIZWIZ [3] and LendAnEye [13], and others [6,[13][14][15]. These solutions make use of the integration between human resource and the information technology.…”

“…Indeed, they can ask anyone for the unknown object but there were some confidential data and situations that he can't share with any strangers except close friend or a family's member [6].…”

Blind-environment interaction through voice augmented objects

“…These descriptors are invariant to translation, scaling and rotation of objects and partially invariant to changes in illumination. Relatively quick calculation of the image features allows the development of systems for object recognition that work nearly in real-time [11]. Computer-vision-based techniques have a higher functionality, but also drawbacks such as: (1) high cost of server-side hardware and software; (2) still low recognition accuracy of such techniques resulting in safety concerns for use by BVI persons [9,12]-recognition depends not only on the descriptor used but also on the training data, training algorithm and type of classifier; (3) such systems are sensitive to the illumination; (4) intensive camera use of mobile devices quickly shortens battery life; (5) intensive network traffic, especially in systems where images are processed entirely by software from the server side, involves paying a higher price for mobile data transfer; and (6) it is still hard to extract detailed descriptions of objects from images.…”