Acoustic Room Modelling Using 360 Stereo Cameras

Abstract

In this paper we propose a pipeline for estimating acoustic 3D room structure with geometry and attribute prediction using spherical 360 • cameras. Instead of setting microphone arrays with loudspeakers to measure acoustic parameters for specific rooms, a simple and practical single-shot capture of the scene using a stereo pair of 360 cameras can be used to simulate those acoustic parameters. We assume that the room and objects can be represented as cuboids aligned to the main axes of the room coordinate (Manhattan world). The scene is captured as a stereo pair using off-the-shelf consumer spherical 360 cameras. A cuboid-based 3D room geometry model is estimated by correspondence matching between captured images and semantic labelling using a convolutional neural network (SegNet). The estimated geometry is used to produce frequency-dependent acoustic predictions of the scene. This is, to our knowledge, the first attempt in the literature to use visual geometry estimation and object classification algorithms to predict acoustic properties. Results are compared to measurements through calculated reverberant spatial audio object parameters used for reverberation reproduction customized to the given loudspeaker set up.

A Texture Superpixel Approach to Semantic Material Classification for Acoustic Geometry Tagging

“…Kim et al [16] introduced the frst approach to geometry estimation using scene understanding inferring acoustic characteristics from visual representations of environments. Their pipeline identifes isotropic features in synthesised directional impulse responses, expressed as independent parameters considering direction, time of arrival and spatial information of the sound source with respect to the listener's position.…”

“…For every label, a one-tomany mapping groups measurements of the given material. Following the methodology in [16], we use median frequency-dependent values to determine acoustic absorption, defning acoustic materials. A single acoustic material maps to each given mesh, associating a vector of acoustic absorption coefcients to its triangles, determining the overall acoustic mapping accuracy to depend upon the mesh separation of the scene geometry.…”

A Composite T60 Regression and Classification Approach for Speech Dereverberation

“…Instead of estimating T 60 directly from reverberant speech itself, we could address the dereverberation problem with visual-only data [58], [59] or audio-video data [60], [61]. For a 3D video, the video will contain the corresponding room environment characteristics, such as room dimensions, the positions of microphone and the speaker, and objects in the room that cause the reflections.…”

An overview of machine learning and other data-based methods for spatial audio capture, processing, and reproduction

“…Training pairs of features of room geometrical configurations and spatial impulse responses captured for those configurations are used in [76]. Alternatively, simplified room geometries are reconstructed from 360 camera images in [77,78], which are used to drive virtual acoustic simulators for AR/VR applications. In the same spirit, [79] uses a DNN to classify materials from textures in a 3D room geometry, to deduce and optimize absorption coefficients to be used in conjunction with the geometry for interactive geometrical acoustics simulations.…”