A Study of Material Sonification in Touchscreen Devices

Abstract

Even in the digital age, designers largely rely on physical material samples to illustrate their products, as existing visual representations fail to sufficiently reproduce the look and feel of real world materials. Here, we investigate the use of interactive material sonification as an additional sensory modality for communicating well-established material qualities like softness, pleasantness or value. We developed a custom application for touchscreen devices that receives tactile input and translate it into material rubbing sound using granular synthesis. We used this system to perform a psychophysical study, in which the ability of the user to rate subjective material qualities is evaluated, with the actual material samples serving as reference stimulus. Our experimental results indicate that the considered audio cues do not significantly contribute to the perception of material qualities but are able to increase the level of immersion when interacting with digital samples.

Does It Ping or Pong? Auditory and Tactile Classification of Materials by Bouncing Events

“…Inner material properties can be actively explored using touch and hearing, as when an indentation or a tap unveil the hardness of an object. Point-wise tapping in particular generates impulsive audio-tactile feedback whose role goes beyond aesthetics [41], giving rise to cross-modal cues which are difficult to disambiguate [17].…”