Toward local and global perception modules for vision substitution

Bologna, Guido; Deville, Benoît; Gomez, Juan Diego; Pun, Thierry

doi:10.1016/j.neucom.2010.07.022

Cited by 12 publications

(9 citation statements)

References 26 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The local module provides the user with the auditory representation of a row containing 25 points of the central part of captured image [35] ( Figure 3). These points are coded into leftright spatialized musical instrument sounds, in order to represent and emphasize the color and location of visual entities in the environment [37], [38], [39].…”

Section: The Local Modulementioning

confidence: 99%

“…Afterwards, a visually impaired user that performs exploration with See ColOr is informed about the presence of learned objects in real time, if any ( Figure 4, left, center). Overall, this module allows the blind noticing serendipitous discoveries, seeking a specific target, as well as avoiding obstacles [35] ( Figure 5). Notice that the mental modeling of a chair becomes easier when the user already knows "this is a chair".…”

Section: The Recognition Modulementioning

confidence: 99%

See 1 more Smart Citation

See ColOr: an extended sensory substitution device for the visually impaired

Gomez

Bologna

Pun

2014

Journal of Assistive Technologies

Self Cite

View full text Add to dashboard Cite

Purpose – The purpose of this paper is to overcome the limitations of sensory substitution methods (SSDs) to represent high-level or conceptual information involved in vision, which are mainly produced by the biological sensory mismatch between sight and substituting senses. Thus, provide the visually impaired with a more practical and functional SSD. Design/methodology/approach – Unlike any other approach, the SSD extends beyond a sensing prototype, by integrating computer vision methods to produce reliable knowledge about the physical world (at the lowest cost to the user). Importantly though, the authors do not abandon the typical encoding of low-level features into sound. The paper simply argues that any visual perception can be achieved through hearing needs to be reinforced or enhanced by techniques that lie beyond mere visual-to-audio mapping (e.g. computer vision, image processing). Findings – Experiments reported in this paper reveal that the See ColOr is learnable and functional, and provides easy interaction. In moderate time, participants were enabled to grasp visual information of the world out of which they could derive: spatial awareness, ability to find someone, location of daily objects and skill to walk safely avoiding obstacles. The encouraging results open a door toward autonomous mobility of the blind. Originality/value – The paper uses the “extended” approach to introduce and justify that the system is brand new, as well as the experimental studies on computer-vision extension of SSDs that are presented. Also, this is the first paper reporting on a terminated, integrated and functional system

show abstract

Section: The Local Modulementioning

confidence: 99%

Section: The Recognition Modulementioning

confidence: 99%

See ColOr: an extended sensory substitution device for the visually impaired

Gomez

Bologna

Pun

2014

Journal of Assistive Technologies

Self Cite

View full text Add to dashboard Cite

show abstract

“…That work was in contrast to existing sonification frameworks which e.g. help visually impaired navigating through environments [25], [3]. There has been done previous work on the sonification of low-level characteristics of images for visual impaired.…”

Section: Introductionmentioning

confidence: 99%

Sonification of images for the visually impaired using a multi-level approach

Banf

Blanz

2013

Proceedings of the 4th Augmented Human International Conference

View full text Add to dashboard Cite

This paper presents a system that strives to give visually impaired persons direct perceptual access to images via an acoustic signal. The user explores the image actively on a touch screen and receives auditory feedback about the image content at the current position. The design of such a system involves two major challenges: what is the most useful and relevant image information, and how can as much information as possible be captured in an audio signal. We address both problems, and propose a general approach that combines low-level information, such as color, edges, and roughness, with mid-and high-level information obtained from Machine Learning algorithms. This includes object recognition and the classification of regions into the categories "man made" versus "natural". We argue that this multi-level approach gives users direct access to what is where in the image, yet it still exploits the potential of recent developments in Computer Vision and Machine Learning.

show abstract

“…Trying to adapt, to a greater or lesser extent, to the world is a constant challenge. Many works have addressed the environment understanding [5,6] and the urban navigation problem [7,8], by developing assistive devices based on smart sensors or artificial vision [9]; some others have proposed solutions for reading [10,11] and computer access [12,13] by exploiting the same devices and technologies. However, few works have focused on assisting visually impaired (VI) people in finding daily used objects.…”

Section: Introductionmentioning

confidence: 99%

Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired

et al. 2020

View full text Add to dashboard Cite

This paper introduces a novel low-cost solar-powered wearable assistive technology (AT) device, whose aim is to provide continuous, real-time object recognition to ease the finding of the objects for visually impaired (VI) people in daily life. The system consists of three major components: a miniature low-cost camera, a system on module (SoM) computing unit, and an ultrasonic sensor. The first is worn on the user’s eyeglasses and acquires real-time video of the nearby space. The second is worn as a belt and runs deep learning-based methods and spatial algorithms which process the video coming from the camera performing objects’ detection and recognition. The third assists on positioning the objects found in the surrounding space. The developed device provides audible descriptive sentences as feedback to the user involving the objects recognized and their position referenced to the user gaze. After a proper power consumption analysis, a wearable solar harvesting system, integrated with the developed AT device, has been designed and tested to extend the energy autonomy in the different operating modes and scenarios. Experimental results obtained with the developed low-cost AT device have demonstrated an accurate and reliable real-time object identification with an 86% correct recognition rate and 215 ms average time interval (in case of high-speed SoM operating mode) for the image processing. The proposed system is capable of recognizing the 91 objects offered by the Microsoft Common Objects in Context (COCO) dataset plus several custom objects and human faces. In addition, a simple and scalable methodology for using image datasets and training of Convolutional Neural Networks (CNNs) is introduced to add objects to the system and increase its repertory. It is also demonstrated that comprehensive trainings involving 100 images per targeted object achieve 89% recognition rates, while fast trainings with only 12 images achieve acceptable recognition rates of 55%.

show abstract

Toward local and global perception modules for vision substitution

Cited by 12 publications

References 26 publications

See ColOr: an extended sensory substitution device for the visually impaired

See ColOr: an extended sensory substitution device for the visually impaired

Sonification of images for the visually impaired using a multi-level approach

Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired

Contact Info

Product

Resources

About