End-to-End Binaural Speech Synthesis

Huang, Wen-Chin; Marković, Dejan; Richard, Alexander; Gebru, Israel D.; Menon, Anjali Kondur

doi:10.21437/interspeech.2022-10603

Cited by 5 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liang et al [48] proposed an audio-visual neural field in real-world audio-visual scenes. Many recent works have proposed to generate spatial audio from mono audio using visual cues [62,28,72,90,49,100,30], or the relative pose between sound sources and the receiver [76,39]. Inspired by these works, our feature learning approach learns spatial representations through an audio prediction task.…”

Section: Related Workmentioning

confidence: 99%

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

Chen¹,

Qian²,

Owens³

2023

Preprint

View full text Add to dashboard Cite

The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https://ificl.github.io/SLfM .

show abstract