Effect of audio-tactile congruence on vibrotactile music enhancement

Abstract

Music listening experiences can be enhanced with tactile vibrations. However, it is not known which parameters of the tactile vibration must be congruent with the music to enhance it. Devices that aim to enhance music with tactile vibrations often require coding an acoustic signal into a congruent vibrotactile signal. Therefore, understanding which of these audio-tactile congruences are important is crucial. Participants were presented with a simple sine wave melody through supra-aural headphones and a haptic actuator held between the thumb and forefinger. Incongruent versions of the stimuli were made by randomizing physical parameters of the tactile stimulus independently of the auditory stimulus. Participants were instructed to rate the stimuli against the incongruent stimuli based on preference. It was found making the intensity of the tactile stimulus incongruent with the intensity of the auditory stimulus, as well as misaligning the two modalities in time, had the biggest negative effect on ratings for the melody used. Future vibrotactile music enhancement devices can use time alignment and intensity congruence as a baseline coding strategy, which improved strategies can be tested against.

Vibrotactile enhancement of musical engagement

“…Due to the wide range of spectro-temporal variability and differences in the average frequency content of the tracks, this approach naturally came along with a lack of control with regards to the vibration stimulus, because we used the low-pass filtered audio signal to directly drive the vibration signal. Recent results (which we were not aware of at the time of designing the experiment) indeed suggest more effective approaches for deriving the vibration stimulus from the audio 19 . Furthermore, we here decided to use one fixed level of vibration in order to be able to interpret individual differences of vibrotactile effects.…”

“…Specifically, the dimensions of a dissimilarity space derived from the ratings of normal-hearing listeners could be accurately modeled by a linear combination of audio and tactile stimulus features (attack time, roughness). Using simple sine wave melodies, it was further observed that time and intensity alignment are most critical for ratings of stimulus preference (used as a proxy of vibrotactile coherence) 19 . Frequency congruence was only observed to play a notable role for musician participants with frequency changes stretched by a factor of 1.5, suggesting a negligible role in terms of practical relevance.…”

Tactile displays for auditory augmentation–A scoping review and reflections on music applications for hearing impaired users

“…Rhythm-the temporal relationship of events in a musical context-is arguably the first aspect to be discussed, as it is fairly well transmitted through tactile channels (Jiam and Limb, 2019). Substantial research has been dedicated to understanding the vibrotactile rhythm, investigating its impact on music aesthetics (Swerdfeger et al, 2009;Hove et al, 2019), and in regards to D/HOH individuals, its enhancement properties (Gilmore and Russo, 2020;Aker et al, 2022) as well as its interaction with the auditory counterpart (Lauzon et al, 2020). When it comes to pitch-the perceived (vibrotactile) frequencythe bandwidth is very limited compared to the ear, but this does not mean that humans cannot perceive pitch differences, only that the just noticeable difference between intervals must be larger, and the range is bound within 20 Hz-1,000 Hz (Chafe, 1993).…”

“…This translates to a lower level of robust synergy between the two sensory apparatuses that can be exploited to synthesize experiences impossible to achieve by unisensory means. Furthermore, research on auditory-tactile interactions has shown that tactile stimulus can influence the auditory stimulus and vice versa (Ro et al, 2009;Okazaki et al, 2012Okazaki et al, , 2013Aker et al, 2022). It can therefore be observed that auditory and haptic stimuli are capable of modifying or altering the perception of each other when presented in unison, as described by Young et al (2016) and Aker et al (2022), and studied extensively with respect to music experiences (Russo, 2019).…”

Vibrotactile enhancement of musical engagement

“…Specifically, the dimensions of a dissimilarity space derived from the ratings of normal-hearing listeners could be accurately modeled by a linear combination of audio and tactile stimulus features (attack time, roughness). Using simple sine wave melodies, it was further observed that time and intensity alignment are most critical for ratings of stimulus preference (used as a proxy of vibrotactile coherence) 22 . Frequency congruence was only observed to play a notable role for musician participants with frequency changes stretched by a factor of 1.5, suggesting a negligible role in terms of practical relevance.…”