Integrated multimodal interaction using texture representations

Sterling, Auston

doi:10.1016/j.cag.2015.10.010

Cited by 4 publications

(2 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This kind of samplebased method has a simple process with high synthesis efficiency, but it needs more parameter control, and the acquisition of parameters often needs manual intervention. Sterling and Lin [204] further improved the method by introducing a vector graph [202] and high-order damping attenuation.…”

Section: Sound Texture Synthesis Methodsmentioning

confidence: 99%

Sound Synthesis, Propagation, and Rendering: A Survey

Liu,

Manocha

2020

Preprint

View full text Add to dashboard Cite

Sound, as a crucial sensory channel, plays a vital role in improving the reality and immersiveness of a virtual environment, following only vision in importance. Sound can provide important clues such as sound directionality and spatial size. This paper gives a broad overview of research works on sound simulation in virtual reality, games, etc. We first survey various sound synthesis methods, including harmonic synthesis, texture synthesis, spectral analysis, and physics-based synthesis. Then, we summarize popular sound propagation techniques, namely wave-based methods, geometric-based methods, and hybrid methods. Next, the sound rendering methods are reviewed. We also highlight some recent methods that use machine learning techniques for synthesis, propagation, and some inverse problems.

show abstract

Section: Sound Texture Synthesis Methodsmentioning

confidence: 99%

Sound Synthesis, Propagation, and Rendering: A Survey

Liu,

Manocha

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…While numerous approaches have been introduced to render vibrotactile feedback and sound independently, limited effort has been directed towards their simultaneous rendering. For instance, Sterling et al 21 exhibited that normal maps and relief maps can serve as integrated representations of intricate surface details and render haptic textures as well as synthesize sounds. In 1 , object geometry and material properties, including displacement and roughness texture maps, are employed within a physics-based model to achieve the realistic rendering of both vibrotactile feedback and sound.…”

Section: Introductionmentioning

confidence: 99%

A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering

Joolee,

Uddin

2024

Sci Rep

View full text Add to dashboard Cite

Data-driven approaches are often utilized to model and generate vibrotactile feedback and sounds for rigid stylus-based interaction. Nevertheless, in prior research, these two modalities were typically addressed separately due to challenges related to synchronization and design complexity. To this end, we introduce a novel multimodal multitask deep learning framework. In this paper, we developed a comprehensive end-to-end data-driven system that encompasses the capture of contact acceleration signals and sound data from various texture surfaces. This framework introduces novel encoder-decoder networks for modeling and rendering vibrotactile feedback through an actuator while routing sound to headphones. The proposed encoder-decoder networks incorporate stacked transformers with convolutional layers to capture both local variability and overall trends within the data. To the best of our knowledge, this is the first attempt to apply transformer-based data-driven approach for modeling and rendering of vibrotactile signals as well as sounds during tool-surface interactions. In numerical evaluations, the proposed framework demonstrates a lower RMS error compared to state-of-the-art models for both vibrotactile signals and sound data. Additionally, subjective similarity evaluation also confirm the superiority of proposed method over state-of-the-art.

show abstract

Foreword to special section on Graphics Interface 2015

Zhang

Tang

2016

Computers & Graphics

View full text Add to dashboard Cite

Integrated multimodal interaction using texture representations

Cited by 4 publications

References 26 publications

Sound Synthesis, Propagation, and Rendering: A Survey

Sound Synthesis, Propagation, and Rendering: A Survey

A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering

Foreword to special section on Graphics Interface 2015

Contact Info

Product

Resources

About