Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution

Abstract

Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.

Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution

“…For example, the authors of [12] used machine learning to examine the behavioral success using auditory encoding of a visual-auditory SSD; they proposed two cross-modal perception models for late-blind and congenitally blind individuals because their exposures to visual stimulation are different. In [30], a cross-modal GANbased evaluation method was proposed to find the optimal auditory sensitivity to reduce the transmission latency in visual-auditory sensory substitution. However, these studies have simply demonstrated the possibility of applying deep learning methods to SSD.…”

“…approach to evaluate the efficiency of the encoding scheme and qualify the visual perception of a visual-auditory SSD. Our approach is similar to that employed in [12], [30] in that it uses vOICe as the encoding method for visual-auditory sensory substitution and creates synthesized visual data from the encoded audio signals by using a cross-modal GAN. However, we extend these approaches to obtain a method that yields an optimized design of the visual-auditory SSD algorithm and objectively quantify the visual perception from the behavioral analysis.…”

Use of Auditory Cues and Other Strategies as Sources of Spatial Information for People with Visual Impairment When Navigating Unfamiliar Environments

“…Various studies have shown that people who are visually impaired show higher levels of auditory ability due to their frequent use in this population. These abilities are pertinent to auditory skills such as echo processing, voice localizing, and distance estimation [ 13 , 14 , 15 , 16 ].…”