Real-time perception-level translation from audio signals to vibrotactile effects

Abstract

In this paper, we propose a real-time perception-level audioto-vibrotactile translation algorithm. Unlike previous signallevel conversion methods, our algorithm considers only perceptual characteristics, such as loudness and roughness, of audio and tactile stimuli. This perception-level approach allows for designing intuitive and explicit conversion models with clear understandings of their perceptual consequences. Our current implementation is tailored to accurate detection of special sound effects to provide well-synchronized audiotactile feedback in immersive applications. We also assessed the performance of our translation algorithm in terms of the detection rate of special sound effects, computational performance, and user preference. All the experimental results supported that our algorithm works well as intended with better performance than the signal-level conversion methods, especially for games. Our algorithm can be easily realized in current products, including mobile devices, gaming devices, and 4D home theater systems, for richer user experience.

Automatic Generation of Spatial Tactile Effects by Analyzing Cross-modality Features of a Video

“…One of the key requirements for generating haptic effects is the synchronization with the audiovisual content. There has been mainly three ways to achieve this: capturing physical sensor data during filming [7], having haptic content designers manually add haptic features [31,37], and automatically generating haptic effects based on the video or audio content [17,21,22].…”

“…For example, these methods will always generate strong tactile stimuli whenever there is a large object moving across the scene. Audio signals have also been used to generate haptic effects by analyzing their frequency characteristics [5] or perception-level intensity [22]. However, simply translating an audio signal to a haptic signal alone without understanding its semantic correspondence with visual content could potentially lead to inadequate haptic effects.…”

“…For instance, generating temporally synchronized tactile effects for a moving truck that honks sporadically is impossible with visual features alone. Audio signals without any visual information (e.g., music or special sound effects) have also been translated to tactile stimuli [5,22]. However, the resulting haptic content is not spatially aligned with the visual content.…”

Toward Haptic Cinematography: Enhancing Movie Experiences with Camera-Based Haptic Effects

“…music, voice-over, etc.). Here haptic effects would highlight particular sound effects or music [9]. In a second category, haptic effects underline the context, i.e.…”

Implementation and Characterization of Vibrotactile Interfaces

“…Among the approaches adopted to design vibrotactile feedback for non-visual information display, complex semantics have been investigated [20] on top of simpler vibrotactile codes [3,22]. Focusing in particular on DMIs, the most straightforward solution is to obtain tactile signals directly from their audio output.…”