Automatic Generation of Spatial Tactile Effects by Analyzing Cross-modality Features of a Video

Abstract

Tactile effects can enhance user experience of multimedia content. However, generating appropriate tactile stimuli without any human intervention remains a challenge. While visual or audio information has been used to automatically generate tactile effects, utilizing cross-modal information may further improve the spatiotemporal synchronization and user experience of the tactile effects. In this paper, we present a pipeline for automatic generation of vibrotactile effects through the extraction of both the visual and audio features from a video. Two neural network models are used to extract the diegetic audio content, and localize a sounding object in the scene. These models are then used to determine the spatial distribution and the intensity of the tactile effects. To evaluate the performance of our method, we conducted a user study to compare the videos with tactile effects generated by our method to both the original videos without any tactile stimuli and videos with tactile effects generated based on visual features only. The study results demonstrate that our cross-modal method creates tactile effects with better spatiotemporal synchronization than the existing visual-based method and provides a more immersive user experience. CCS CONCEPTS • Human-centered computing → Haptic devices.

Evaluation of Multimodal External Human–Machine Interface for Driverless Vehicles in Virtual Reality

“…These vehicle behaviors probably effect pedestrians' anticipation of the vehicle's intentions. Other scholars [20,27] suggested that physical modality can also be a kind of haptic feedback, such as a mobile phone vibration [28] or haptic actuator. However, our goal is to design vehicle-based eHMI which means these interfaces involve placing cues on the vehicle only.…”

AVscript: Accessible Video Editing with Audio-Visual Scripts

“…The visualization community has demonstrated ample applications that support multimodal data exploration with touch and speech (e.g., [35,[74][75][76]). In similar vein, we plan to build on work in tactile displays [85] to surface the visual content in the video. While consuming video with tactile displays may be challenging, editing video may benefit from providing creators access to slow frame-by-frame content (e.g., to assess when a person moves out of the frame) and waveform visualizations.…”