Tactile effects can enhance user experience of multimedia content. However, generating appropriate tactile stimuli without any human intervention remains a challenge. While visual or audio information has been used to automatically generate tactile effects, utilizing cross-modal information may further improve the spatiotemporal synchronization and user experience of the tactile effects. In this paper, we present a pipeline for automatic generation of vibrotactile effects through the extraction of both the visual and audio features from a video. Two neural network models are used to extract the diegetic audio content, and localize a sounding object in the scene. These models are then used to determine the spatial distribution and the intensity of the tactile effects. To evaluate the performance of our method, we conducted a user study to compare the videos with tactile effects generated by our method to both the original videos without any tactile stimuli and videos with tactile effects generated based on visual features only. The study results demonstrate that our cross-modal method creates tactile effects with better spatiotemporal synchronization than the existing visual-based method and provides a more immersive user experience. CCS CONCEPTS • Human-centered computing → Haptic devices.

In this paper, we focus on the visual features of steam injection and propose an integrated algorithm to detect it based on video surveillance. The proposed method is depended on three decision rules which are the attribute of gray level and the feature of frequent flicker rate of steam injection, and the similarity structure between background image and current frame. The block-based approach is applied to all three decision rules. The experimental results show that the algorithm provides a reliable detection method which is useful in many cases such as the alarm on the leakage of a heating pipe.