Integrating physically based sound models in a multimodal rendering architecture

Avanzini, Federico; Crosato, Paolo

doi:10.1002/cav.144

Cited by 24 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The possibility of investigating the interaction between auditory and haptic feedback has been facilitated by the rapid progress of haptic technology, together with the development of efficient and accurate simulation algorithms. However, research on the interaction between touch and audition has focused mainly on hand-based interactions [8], [9], [10], while few studies have been conducted on the interaction of these two sensory modalities in the feet. One exception is the work of Giordano et al, who showed that the feet are effective at probing the world with discriminative touch, with and without access to auditory information [11].…”

Section: Introductionmentioning

confidence: 99%

Haptic Feedback for Enhancing Realism of Walking Simulations

Turchet

Burelli

Serafin

2013

IEEE Trans. Haptics

View full text Add to dashboard Cite

In this paper, we describe several experiments whose goal is to evaluate the role of plantar vibrotactile feedback in enhancing the realism of walking experiences in multimodal virtual environments. To achieve this goal we built an interactive and a noninteractive multimodal feedback system. While during the use of the interactive system subjects physically walked, during the use of the noninteractive system the locomotion was simulated while subjects were sitting on a chair. In both the configurations subjects were exposed to auditory and audio-visual stimuli presented with and without the haptic feedback. Results of the experiments provide a clear preference toward the simulations enhanced with haptic feedback showing that the haptic channel can lead to more realistic experiences in both interactive and noninteractive configurations. The majority of subjects clearly appreciated the added feedback. However, some subjects found the added feedback unpleasant. This might be due, on one hand, to the limits of the haptic simulation and, on the other hand, to the different individual desire to be involved in the simulations. Our findings can be applied to the context of physical navigation in multimodal virtual environments as well as to enhance the user experience of watching a movie or playing a video game.

show abstract

Section: Introductionmentioning

confidence: 99%

Haptic Feedback for Enhancing Realism of Walking Simulations

Turchet

Burelli

Serafin

2013

IEEE Trans. Haptics

View full text Add to dashboard Cite

show abstract

“…Whilst most studies focus on the interaction between vision and audition or between vision and touch, interaction between touch and audition is also strong because of two sources of sensory information have high temporal resolution. The perception literature contains many reports of audio-tactile interaction effects, see [15,14,21,4] for examples and surveys, and there has been studies directed at leveraging audiotactile to enhance interaction with virtual worlds [19,7,6,17,22,1].…”

Section: Introductionmentioning

confidence: 99%

Preliminary Experiment Combining Virtual Reality Haptic Shoes and Audio Synthesis

Nordahl

Berrezag

Dimitrov

et al. 2010

Haptics: Generating and Perceiving Tangible Sensations

View full text Add to dashboard Cite

Abstract. We describe a system that can provide combined auditory and haptic sensations that arise while walking on different grounds. The simulation is based on a physical model that drives both haptic transducers embedded in sandals and headphones. The model is able to represent walking interactions with solid surfaces that can creak, be covered with crumpling material. The simulation responds to pressure on the floor by a vibrotactile signal felt by the feet. In a preliminary discrimination experiment, 15 participants were asked to recognize four different surfaces in a list of sixteen possibilities and under three different conditions, haptics only, audition only and combined haptic-audition. The results indicate that subjects are able to recognize most of the stimuli in the audition only condition, and some of the material properties such as hardness in the haptics only condition. The combination of auditory and haptic cues does not significantly improve recognition.

show abstract

“…The influence of auditory material-related information on haptic perception has been studied in the domain of virtual reality and ecological perception (see a thorough review by Giordano and Avanzini [36]). The ratings for the stiffness of audio-haptic events were usually found to increase with the auditory stiffness of real [15] and synthesized [16] sounds. By adjusting low-level acoustic parameters while maintaining the material-related information, we found that the auditory effect was not always consistent in the audio-only and audio-haptic conditions.…”

Section: Discussionmentioning

confidence: 99%

Metal Sounds Stiffer than Drums for Ears, but Not Always for Hands: Low-Level Auditory Features Affect Multisensory Stiffness Perception More than High-Level Categorical Information

Liu

Ando

2016

PLoS ONE

View full text Add to dashboard Cite

Most real-world events stimulate multiple sensory modalities simultaneously. Usually, the stiffness of an object is perceived haptically. However, auditory signals also contain stiffness-related information, and people can form impressions of stiffness from the different impact sounds of metal, wood, or glass. To understand whether there is any interaction between auditory and haptic stiffness perception, and if so, whether the inferred material category is the most relevant auditory information, we conducted experiments using a force-feedback device and the modal synthesis method to present haptic stimuli and impact sound in accordance with participants’ actions, and to modulate low-level acoustic parameters, i.e., frequency and damping, without changing the inferred material categories of sound sources. We found that metal sounds consistently induced an impression of stiffer surfaces than did drum sounds in the audio-only condition, but participants haptically perceived surfaces with modulated metal sounds as significantly softer than the same surfaces with modulated drum sounds, which directly opposes the impression induced by these sounds alone. This result indicates that, although the inferred material category is strongly associated with audio-only stiffness perception, low-level acoustic parameters, especially damping, are more tightly integrated with haptic signals than the material category is. Frequency played an important role in both audio-only and audio-haptic conditions. Our study provides evidence that auditory information influences stiffness perception differently in unisensory and multisensory tasks. Furthermore, the data demonstrated that sounds with higher frequency and/or shorter decay time tended to be judged as stiffer, and contact sounds of stiff objects had no effect on the haptic perception of soft surfaces. We argue that the intrinsic physical relationship between object stiffness and acoustic parameters may be applied as prior knowledge to achieve robust estimation of stiffness in multisensory perception.

show abstract

Integrating physically based sound models in a multimodal rendering architecture

Cited by 24 publications

References 24 publications

Haptic Feedback for Enhancing Realism of Walking Simulations

Haptic Feedback for Enhancing Realism of Walking Simulations

Preliminary Experiment Combining Virtual Reality Haptic Shoes and Audio Synthesis

Metal Sounds Stiffer than Drums for Ears, but Not Always for Hands: Low-Level Auditory Features Affect Multisensory Stiffness Perception More than High-Level Categorical Information

Contact Info

Product

Resources

About