Acoustic Classification and Optimization for Multi-Modal Rendering of Real-World Scenes

Schissler, Carl; Loftin, Christian; Manocha, Dinesh

doi:10.1109/tvcg.2017.2666150

Cited by 47 publications

(45 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analytical gradients can significantly accelerate the optimization process. With similar optimization objectives, it was shown that additional gradient information can boost the speed by a factor of over ten times [33,50]. The speed gain shown by Li et al [33] is impressive, and we further improve the accuracy and speed of the formulation.…”

Section: Related Workmentioning

confidence: 64%

“…We also achieve faster optimization speed. Note that the input audio to our method is already noisy and reverberant, whereas [33] [50]. In the highlighted region, we are able to better reproduce the fast decay in the highfrequency range, closely matching the recorded sound.…”

Section: Comparisonsmentioning

confidence: 72%

“…To explain the motivation of our approach, we briefly elaborate on the most difficult parts of previous approaches, upon which our method improves. Previous methods require an impulse response of the environment to estimate acoustic properties [33,50]. Recording an impulse response is a non-trivial task.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Fig. 1: Given a natural sound in a real-world room that is recorded using a cellphone microphone (left), we estimate the acoustic material properties and the frequency equalization of the room using a novel deep learning approach (middle). We use the estimated acoustic material properties for generating plausible sound effects in the virtual model of the room (right). Our approach is general and robust, and works well with commodity devices.Abstract-We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

show abstract

Section: Related Workmentioning

confidence: 64%

Section: Comparisonsmentioning

confidence: 72%

See 1 more Smart Citation

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Tang

Bryan

et al. 2020

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

show abstract

“…We now estimate the acoustic material from the reconstructed image using the widely used CNN to predict visual appearance. As mentioned in [7], [8], estimating the visual aspect of an image can be considered as a kind of semantic segmentation. In general, semantic segmentation tasks classify and separate various objects in the scene.…”

Section: B Acoustic Materials Estimationmentioning

confidence: 99%

“…Recent SSL papers considering reflection [5] and diffraction [6] only use the 3-D geometry of indoor rooms for the real-time processing, without taking into account acoustic material information. Because accurate estimation of acoustic material is difficult, several approaches [7], [8] were proposed for the approximate estimation using visual appearance without sound information. These methods use RGB or RGB-D images to perform the acoustic material estimation and the 3-D reconstructions.…”

Section: Introductionmentioning

confidence: 99%

Real-time 3-D Mapping with Estimating Acoustic Materials

Kim

Kwon

Yoon

2020

2020 IEEE/SICE International Symposium on System Integration (SII)

View full text Add to dashboard Cite

This paper proposes a real-time system integrating an acoustic material estimation from visual appearance and an on-the-fly mapping in the 3-dimension. The proposed method estimates the acoustic materials of surroundings in indoor scenes and incorporates them to a 3-D occupancy map, as a robot moves around the environment. To estimate the acoustic material from the visual cue, we apply the state-of-the-art semantic segmentation CNN network based on the assumption that the visual appearance and the acoustic materials have a strong association. Furthermore, we introduce an update policy to handle the material estimations during the online mapping process. As a result, our environment map with acoustic material can be used for sound-related robotics applications, such as sound source localization taking into account various acoustic propagation (e.g., reflection).

show abstract

Inexpensive indoor acoustic material estimation for realistic sound propagation

Kim

Yoon

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

For the realistic and immersive experience in a virtual environment, it is important to estimate and reflect the acoustic characteristic of the real indoor scenes. This article proposes a method of directly measuring the reflection coefficient of a surface, which is an acoustic characteristics in the real environment. Because expensive optimization‐based studies that mainly aim to reproduce recorded sounds indirectly estimate acoustic materials, new estimates are required whenever the actual environment changes. Our approach utilizes the method of the acoustics field to enable anyone to easily and directly measure the reflection coefficient of a real environment and generate sound in a virtual environment. We obtain the impulse response (IR) for the target surface, separate the direct sound and the reflected sound, and calculate the reflection coefficient for each surface. The measurement of the IR and the reflection coefficient calculation takes about 2 s. Our result produces a sound with a similar reverberation time to the sound recorded in the real environment.

show abstract

Acoustic Classification and Optimization for Multi-Modal Rendering of Real-World Scenes

Cited by 47 publications

References 34 publications

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Scene-Aware Audio Rendering via Deep Acoustic Analysis

Real-time 3-D Mapping with Estimating Acoustic Materials

Inexpensive indoor acoustic material estimation for realistic sound propagation

Contact Info

Product

Resources

About