Audio-Material Reconstruction for Virtualized Reality Using a Probabilistic Damping Model

Sterling, Auston; Rewkowski, Nicholas; Klatzky, Roberta L.

doi:10.1109/tvcg.2019.2898822

Cited by 12 publications

(6 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many techniques are based on the use of 3D objects' pre-computed eigenmatrix to perform runtime sound synthesis, e.g., using the parallel computing capabilities of the GPU to speedup the computation [45] or reduce the computational complexity by using approximations [3]. Other methods use more complex modal sound synthesis models to simulate sounds more accurately, including knocking, sliding, and friction on 3D objects [42], acceleration noise synthesis [6], accurate damping models [39] and contact models [47].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

NeuralSound: Learning-based Modal Sound Synthesis With Acoustic Transfer

Jin¹,

Li²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a novel learning-based approach to compute the eigenmodes and acoustic transfer data for the sound synthesis of arbitrary solid objects. Our approach combines two network-based solutions to formulate a complete learning-based 3D modal sound model. This includes a 3D sparse convolution network as the eigendecomposition solver and an encoder-decoder network for the prediction of the Far-Field Acoustic Transfer maps (FFAT Maps). We use our approach to compute the vibration modes (eigenmodes) and FFAT maps for each mode (acoustic data) for arbitrary-shaped objects at interactive rates without any precomputed dataset for any new object. Our experimental results demonstrate the effectiveness and benefits of our approach. We compare its accuracy and efficiency with physically-based sound synthesis methods.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Since the human auditory system cannot directly hear an object's vibration, acoustic transfer models are typically used to describe the sound pressure at a listener's position. There is considerable research on improving the performance of these methods in terms of quality or runtime performance [45,3,47,6,38,39].…”

Section: Introductionmentioning

confidence: 99%

NeuralSound: Learning-based Modal Sound Synthesis With Acoustic Transfer

Jin¹,

Li²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly, Ren et al optimized linear modal analysis parameters to match the given recordings [44]. A probabilistic damping model for audio-material reconstruction has been presented for VR applications [57]. Unlike all previous methods which require a clean IR recording for accurate estimation and optimization of boundary materials, we infer typical material parameters including T 60 values and equalization from raw speech signals using a CNN-based model.…”

Section: Related Workmentioning

confidence: 99%

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Fig. 1: Given a natural sound in a real-world room that is recorded using a cellphone microphone (left), we estimate the acoustic material properties and the frequency equalization of the room using a novel deep learning approach (middle). We use the estimated acoustic material properties for generating plausible sound effects in the virtual model of the room (right). Our approach is general and robust, and works well with commodity devices.Abstract-We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

show abstract

“…This method, grounded on auditory perception [170], can produce realistic-sounding results synchronized with different materials and dynamics. Sterling et al [203] introduced an improved method using probabilistic damping model for extracting material parameters from recorded audio. In order to improve the efficiency of modal sound synthesis, some simplified (modal) sound models have been proposed.…”

Section: Rigid-body Soundmentioning

confidence: 99%

“…In modal sound synthesis for rigid-body objects, it is necessary to evaluate material parameters that can approximate real ones. Sterling et al [203] proposed estimating the damping parameters of materials from recorded impact sounds, greatly alleviating the human effort.…”

Section: Rigid-body Soundmentioning

confidence: 99%

Sound Synthesis, Propagation, and Rendering: A Survey

Liu,

Manocha

2020

Preprint

View full text Add to dashboard Cite

Sound, as a crucial sensory channel, plays a vital role in improving the reality and immersiveness of a virtual environment, following only vision in importance. Sound can provide important clues such as sound directionality and spatial size. This paper gives a broad overview of research works on sound simulation in virtual reality, games, etc. We first survey various sound synthesis methods, including harmonic synthesis, texture synthesis, spectral analysis, and physics-based synthesis. Then, we summarize popular sound propagation techniques, namely wave-based methods, geometric-based methods, and hybrid methods. Next, the sound rendering methods are reviewed. We also highlight some recent methods that use machine learning techniques for synthesis, propagation, and some inverse problems.

show abstract

Audio-Material Reconstruction for Virtualized Reality Using a Probabilistic Damping Model

Cited by 12 publications

References 32 publications

NeuralSound: Learning-based Modal Sound Synthesis With Acoustic Transfer

NeuralSound: Learning-based Modal Sound Synthesis With Acoustic Transfer

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Sound Synthesis, Propagation, and Rendering: A Survey

Contact Info

Product

Resources

About