Transforming 3D Coloured Pixels into Musical Instrument Notes for Vision Substitution Applications

Abstract

The goal of the See ColOr project is to achieve a noninvasive mobility aid for blind users that will use the auditory pathway to represent in real-time frontal image scenes. We present and discuss here two image processing methods that were experimented in this work: image simplification by means of segmentation, and guiding the focus of attention through the computation of visual saliency. A mean shift segmentation technique gave the best results, but for real-time constraints we simply implemented an image quantification method based on the HSL colour system. More particularly, we have developed two prototypes which transform HSL coloured pixels into spatialised classical instrument sounds lasting for 300 ms. Hue is sonified by the timbre of a musical instrument, saturation is one of four possible notes, and luminosity is represented by bass when luminosity is rather dark and singing voice when it is relatively bright. The first prototype is devoted to static images on the computer screen, while the second has been built up on a stereoscopic camera which estimates depth by triangulation. In the audio encoding, distance to objects was quantified into four duration levels. Six participants with their eyes covered by a dark tissue were trained to associate colours with musical instruments and then asked to determine on several pictures, objects with specific shapes and colours. In order to simplify the protocol of experiments, we used a tactile tablet, which took the place of the camera. Overall, colour was helpful for the interpretation of image scenes. Moreover, preliminary results with the second prototype consisting in the recognition of coloured balloons were very encouraging. Image processing techniques such as saliency could accelerate in the future the interpretation of sonified image scenes.

On the use of the auditory pathway to represent image scenes in real-time

“…Another issue is whether it is possible to interpret pictures. Several experiments were carried out by six participants having their eyes enclosed by a dark tissue, and listening to the sounds via headphones [7,8]. In these experiments we use the T3 tactile tablet (cf.…”

“…General targeted applications are the search for items of particular interest for blind users, the manipulation of objects and the navigation in an unknown environment. The See ColOr interface encodes coloured pixels by spatialised musical instrument sounds, in order to represent and emphasize the color and location of visual entities in their environment [6,7,8]. The basic idea is to represent a pixel as a directional sound source with depth estimated by stereo-vision.…”

A perceptual interface for vision substitution in a color matching experiment

“…The purpose is to verify the hypothesis that sounds from musical instruments provide an alternative way to vision of obtaining color information from the environment. For the average participant it is difficult to learn the associations between colors and sounds in just one training session as shown by previous work [6]. However, our participants are not asked to identify colors, but just to pair off the same socks.…”

“…In previous work of the See ColOr project [5], [6], we performed several experiments with six blindfolded persons who were trained to associate colors with musical instruments. In order to simplify the experiments, the participants were asked to identify major components of static pictures presented on a special paper lying on a tactile tablet representing pictures with embossed edges.…”

“…Our See ColOr interface encodes colored pixels by musical instrument sounds, in order to emphasize colored entities of the environment [5], [6]. For instance, when one looks above the horizon and it "sounds" blue using a piano sound, it will be very likely to be the sky.…”