The Sound of the Smell (and taste) of my Shoes too

Abstract

This work discusses basic human senses: sight; sound; touch; taste; and smell; and the way in which it may be possible to compensate for lack of one, or more, of these by explicitly representing stimuli using the remaining senses. There may be many situations or scenarios where not all five of these base senses are being stimulated, either because of an optional restriction or deficit or because of a physical or sensory impairment such as loss of sight or touch sensation. Related to this there are other scenarios where sensory matching problems may occur. For example: a user immersed in a virtual environment may have a sense of smell from the real world that is unconnected to the virtual world. In particular, this paper is concerned with how sound can be used to compensate for the lack of other sensory stimulation and vice-versa. As a link is well established already between the visual, touch, and auditory systems, more attention is given to taste and smell, and their relationship with sound. This work presents theoretical concepts, largely oriented around mapping other sensory qualities to sound, based upon existing work in the literature and emerging technologies, to discuss where particular gaps currently exist, how emotion could be a medium to cross-modal representations, and how these might be addressed in future research. It is postulated that descriptive qualities, such as timbre or emotion, are currently the most viable routes for further study and that this may be later integrated with the wider body of research into sensory augmentation. CCS Concepts • Human-centered computing~HCI theory, concepts and models • Human-centered computing~Accessibility theory, concepts and paradigms • Human-centered computing~Mixed / augmented reality • Human-centered computing~Interaction design theory, concepts and paradigms

Sensory translation between audition and vision

“…Hartshorne, 1934 ; Spence, 2020a ; von Helmholtz, 1878/1971 ; though see also Marks, 2011 ). Ultimately, therefore, it can be argued that the best that one can hope for as far as matching, or translating between, the senses is to use emotionally mediated correspondences (e.g., Cunningham & Weinel, 2016 ; Hauck et al, 2022 ; Spence, 2020a ; Spence & Di Stefano, 2022a , for a review). Evidence showing that basic emotions are recognized in musical stimuli across cultures (Fritz et al, 2009 ) would at least partially support the alleged universality of some elements of emotion-mediated translation from audition and vision.…”

Searching for perceptual similarity within, and between, the (chemical) senses

“…Note, though, that in the latter case, the color stimuli that participants had to choose between varied in terms of a range of dimensions (i.e., hue, saturation, and lightness). Certainly, the notion of emotionally mediated crossmodal correspondences has become increasingly popular in recent years (see Cunningham & Weinel, 2016 ; Hauck et al, 2022 ; Spence, 2020a ; Spence & Di Stefano, 2022 ).…”