Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli

Abstract

Millions of people are blind worldwide. Sensory substitution (SS) devices (e.g., vOICe) can assist the blind by encoding a video stream into a sound pattern, recruiting visual brain areas for auditory analysis via crossmodal interactions and plasticity. SS devices often require extensive training to attain limited functionality. In contrast to conventional attention-intensive SS training that starts with visual primitives (e.g., geometrical shapes), we argue that sensory substitution can be engaged efficiently by using stimuli (such as textures) associated with intrinsic crossmodal mappings. Crossmodal mappings link images with sounds and tactile patterns. We show that intuitive SS sounds can be matched to the correct images by naive sighted participants just as well as by intensively-trained participants. This result indicates that existing crossmodal interactions and amodal sensory cortical processing may be as important in the interpretation of patterns by SS as crossmodal plasticity (e.g., the strengthening of existing connections or the formation of new ones), especially at the earlier stages of SS usage. An SS training procedure based on crossmodal mappings could both considerably improve participant performance and shorten training times, thereby enabling SS devices to significantly expand blind capabilities.

Listen to the Image

“…However, cross-modal plasticity provides the possibility to effectively sense concrete visual content via the audio channel [18,36], which depends on specific image-to-sound translation rules. In practice, before training the blinds to "see" objects with the vOICe device, they should learn the translation rules firstly by identifying simple shapes, which could make them sense the objects more precisely [33]. In other words, the cross-modal translation helps to bridge the visual and audio perception in the brain.…”

“…Different from the simple reversal of the encoding direction [33], we aim to explore more possibilities in optimizing the primary scheme. First of all, a well-known fact is that there exist large differences in bandwidth between vision and audition [18].…”

“…The blind participants' feedback or task performance is usually served as the indicator to the quality of encoding schemes in the conventional assessment [3,33]. Following the conventional evaluation strategy [33], 9 participants are randomly divided into three groups (3 participants per group). Each group corresponds to one of the three encoding schemes.…”

“…Similar results are observed on F1-score. According to the conventional criterion [33], it can be concluded that the introduced modifications indeed help to improve the quality of cross-modal translation. And it further confirms the assumptions about audio length and the characteristics of human hearing.…”

“…Second, amounts of control experiments are required to provide convincing assessments, which therefore need much efforts as well as huge employment cost. As a result, there is few works focusing on the optimization of current encoding scheme, while it indeed plays a crucial role in improving the visual perception ability of the blinds [33,18].…”