A Texture Superpixel Approach to Semantic Material Classification for Acoustic Geometry Tagging

Abstract

The current state of audio rendering algorithms allows efcient sound propagation, refecting realistic acoustic properties of real environments. Among factors afecting realism of acoustic simulations is the mapping between an environment's geometry, and acoustic information of materials represented. We present a pipeline to infer material characteristics from their visual representations, providing an automated mapping. A trained image classifer estimates semantic material information from textured meshes mapping predicted labels to a database of measured frequency-dependent absorption coefcients; trained on a material image patches generated from superpixels, it produces inference from meshes, decomposing their unwrapped textures. The most frequent label from predicted texture patches determines the acoustic material assigned to the input mesh. We test the pipeline on a real environment, capturing a conference room and reconstructing its geometry from point cloud data. We estimate a Room Impulse Response (RIR) of the virtual environment, which we compare against a measured counterpart. CCS CONCEPTS• Computing methodologies → Image processing; Rendering; Neural networks; Simulation evaluation.

Acoustic Rendering Based on Geometry Reduction and Acoustic Material Classification

“…The network, pretrained on ImageNet [33], learns on 32x32 pixel resolution image patches that we extract from appearances sampled from OpenSurfaces, assembling a dataset of about 13M images, split into 9M and 4M train and evaluation sets, respectively. The model converges in 45 epochs, achieves validation accuracy of about 0.83, and, as shown in recent experiments [26], it can replace humans in the process of acoustic material tagging in enclosed scenes of real-world space. Via a one-to-many mapping, we associate each of the 34 categories from the OpenSurfaces dataset to acoustic materials.…”

“…In-the-wild datasets, such as OpenSurfaces [24], and Mat-terport3D [25], capture material appearances in surfaces with sub-optimal lighting conditions, specular reflections and errors introduced by optic sensors, which may result in noisy image representations of materials. Classification of image patches through densely connected networks can achieve reasonable accuracy when trained on extracted features, with the condition that the data has a comprehensive representation of materials applicable to target scenes of the acoustic rendering pipeline, as shown in recent work by Colombo et al [26]. Based on captured real environments, the authors automated the mapping between the visual representation of materials, expressed as meshes whose textures are decomposed into superpixels to infer their acoustic characteristics via a learned classifier.…”

“…This last layer classifies predicted embeddings into acoustic material categories. The network was trained on the OpenSurfaces dataset [24], learning mappings between visual appearances of surfaces from 34 material classes and semantic labels, as described in [26]. The network, pretrained on ImageNet [33], learns on 32x32 pixel resolution image patches that we extract from appearances sampled from OpenSurfaces, assembling a dataset of about 13M images, split into 9M and 4M train and evaluation sets, respectively.…”