Implicit HRTF Modeling Using Temporal Convolutional Networks

Gebru, Israel D.; Marković, Dejan; Richard, Alexander; Krenn, Steven; Butler, Gladstone Alexander; Torre, Fernando De la; Sheikh, Yaser

doi:10.1109/icassp39728.2021.9414750

Cited by 15 publications

(6 citation statements)

References 23 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hyper-convolution layer is a linear module of which the weights and biases are generated from given conditions. In the work by [25,19], the hyper-convolution was used for conditioning the temporal convolutional network that operates in the time domain. We adopt a similar setting with [25], yet without dilation and different kernel size, since our HyperConv deals with encoded positions and anthropometric features.…”

Section: Network Architecturementioning

confidence: 99%

See 1 more Smart Citation

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features

Lee¹,

Lee²,

Lee³

2022

Preprint

View full text Add to dashboard Cite

Estimating Head-Related Transfer Functions (HRTFs) of arbitrary source points is essential in immersive binaural audio rendering. Computing each individual's HRTFs is challenging, as traditional approaches require expensive time and computational resources, while modern data-driven approaches are data-hungry. Especially for the data-driven approaches, existing HRTF datasets differ in spatial sampling distributions of source positions, posing a major problem when generalizing the method across multiple datasets. To alleviate this, we propose a deep learning method based on a novel conditioning architecture. The proposed method can predict an HRTF of any position by interpolating the HRTFs of known distributions. Experimental results show that the proposed architecture improves the model's generalizability across datasets with various coordinate systems. Additional demonstrations using coarsened HRTFs demonstrate that the model robustly reconstructs the target HRTFs from the coarsened data.

show abstract

Section: Network Architecturementioning

confidence: 99%

“…Recently, data-driven methods for binaural synthesis have been widely studied [16,17,18,19]. These methods inherently require a sufficiently large number of data points to estimate the distribution of interest.…”

Section: Introductionmentioning

confidence: 99%

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features

Lee¹,

Lee²,

Lee³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Binaural audio synthesis has traditionally relied on signal processing techniques that model the physics of human spatial hearing as a linear time-invariant system [17][18][19][20]. More recently, there has been a line of studies on neural synthesis of binaural audio that showed the advantages of the data-driven approaches [2][3][4][21][22][23][24][25]. We will refer to these models, illustrated at the top right of Fig.…”

Section: Related Workmentioning

confidence: 99%

“…Although binaural synthesis has recently experienced a breakthrough based on neural audio rendering techniques [2][3][4] that allow to learn binauralization and spatial audio in a datadriven way, these approaches fall short in their ability to faithfully model environmental factors such as room reverb and noise floor. The reason these models fail to model stochastic processes is their reliance on direct reconstruction losses on waveforms.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Binaural Speech Synthesis

Huang¹,

Marković²,

Richard³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we present an end-to-end binaural speech synthesis system that combines a low-bitrate audio codec with a powerful binaural decoder that is capable of accurate speech binauralization while faithfully reconstructing environmental factors like ambient noise or reverb. The network is a modified vectorquantized variational autoencoder, trained with several carefully designed objectives, including an adversarial loss. We evaluate the proposed system on an internal binaural dataset with objective metrics and a perceptual study. Results show that the proposed approach matches the ground truth data more closely than previous methods. In particular, we demonstrate the capability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.

show abstract

“…In addition, this paper investigates an end-to-end neural network synthesis method that simulates the differences caused by subtle effects on the final output signal through a temporal convolutional neural network module. We conducted extensive experiments on the POP909 dataset [7] and HRFT data [8] to evaluate the effectiveness and generality of the proposed method.…”

Section: Introductionmentioning

confidence: 99%

MetaMGC: a music generation framework for concerts in metaverse

Jin

Wang

et al. 2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

In recent years, there has been a national craze for metaverse concerts. However, existing meta-universe concert efforts often focus on immersive visual experiences and lack consideration of the musical and aural experience. But for concerts, it is the beautiful music and the immersive listening experience that deserve the most attention. Therefore, enhancing intelligent and immersive musical experiences is essential for the further development of the metaverse. With this in mind, we propose a metaverse concert generation framework — from intelligent music generation to stereo conversion and sound field design for virtual concert stages. First, combining the ideas of reinforcement learning and value functions, the Transformer-XL music generation network is improved and used in training all the music in the POP909 dataset. Experiments show that both improved algorithms have advantages over the original method in terms of objective evaluation and subjective evaluation metrics. In addition, this paper validates a neural rendering method that can be used to generate spatial audio based on a binaural-integrated neural network with a fully convolutional technique. And the purely data-driven end-to-end model performs to be more reliable compared with traditional spatial audio generation methods such as HRTF. Finally, we propose a metadata-based audio rendering algorithm to simulate real-world acoustic environments.

show abstract

Implicit HRTF Modeling Using Temporal Convolutional Networks

Cited by 15 publications

References 23 publications

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features

Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features

End-to-End Binaural Speech Synthesis

MetaMGC: a music generation framework for concerts in metaverse

Contact Info

Product

Resources

About