This paper presents a multimodal rendering architecture that integrates physically based sound models with haptic and visual rendering. The proposed sound modeling approach is compared to other existing techniques. An example of implementation of the architecture is presented, that realizes bimodal (auditory and haptic) rendering of contact stiffness. It is shown that the proposed rendering scheme allows tight synchronization of the two modalities, as well as a high degree of interactivity and responsiveness of the sound models to gestures and actions of a user. Finally, an experiment on the relative contributions of haptic and auditory information to bimodal judgments of contact stiffness is presented. Experimental results support the effectiveness of auditory feedback in modulating haptic perception of stiffness
Many theories of multisensory integration focus on action-independent perception. This approach has limited ecological validity, because under everyday conditions we frequently act upon multisensory events through motor behavior. We present a novel methodology for the study of multisensory integration in action-directed perception, focusing on striking events. Observers repeatedly hit a virtual object with a target striking velocity, and are presented with various types of sensory information about the striking event: auditory and/or haptic and/or visual. For each of the experimental trials, they initially receive feedback on the tracking of the target striking velocity. In a second phase, feedback is eliminated. In a third phase, the sensory properties of the striking event are changed. We quantify the extent to which a variation in each of the sensory modalities disrupts performance in the tracking of the target striking velocity. Multisensory dominance hierarchies are inferred from the comparison of the patterns of performance disruption in unimodal and multisensory contexts.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.