Software Design for Low-Latency Visuo-Auditory Sensory Substitution on Mobile Devices

Abstract

Visuo-auditory sensory substitution devices transform a video stream into an audio stream to help visually impaired people in situations where spatial information is required, such as avoiding moving obstacles. In these particular situations, the latency between an event in the real world and its auditory transduction is of paramount importance. In this article, we describe an optimized software architecture for low-latency video-to-audio transduction using current mobile hardware. We explain step-by-step the required computations and we report the corresponding measured latencies. The whole latency is approximately 65 ms with a capture resolution of 160 × 120 at 30 frames-per-second and 1000 sonified pixels per frame.

Visual-auditory substitution device for indoor navigation based on fast visual marker detection

“…Nearby obstacles are extracted from the depth video stream of the RGB-D camera with a threshold of elements located in the visual scene within one meter of the user. Our sonification method is based on a low latency auditory substitution approach [21] where each visual position is associated with a short spatialized stereophonic audio pixel. A spatialize sound allows a precise localization of an area of interest without emitting a long verbal expression which increases the delay between two successive information and thus increases the danger of navigation.…”

Low-Latency Human-Computer Auditory Interface Based on Real-Time Vision Analysis

“…More precisely, we aim to minimize the delay between an image acquisition and the sound transmission to the user. The Li-breAudioView sonification architecture has been optimized in a previous paper [11]: the optimization of the sonification software has reduced the required processing time by 86% compared to the original version [3]. After the optimization of the sound generation stage, the image processing part represents 95% of the global processing time on a standard PC platform (Intel Core i7-6700HQ processor : 4 Cores -8 Threads, 2.60 GHz; 16 GB RAM).…”