Seeing through Sounds: Predicting Visual Semantic Segmentation Results from Multichannel Audio Signals

Irie, Go; Ostrek, Mirela; Wang, Haochen; Kameoka, Hirokazu; Kimura, Atsushi; Kawanishi, Takahito; Kashino, Kunio

doi:10.1109/icassp.2019.8683142

Cited by 18 publications

(20 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, Binaural sounds are different from general stereo sounds. Another similar work to ours is done by Irie et al [52]. While their goal is also to predict semantic segmentation results from multichannel audio signals, our method differs significantly: 1) our method considers multiple semantic classes, 2) our method works in unconstrained real outdoor environment instead of a controlled lab setting, and 3) our method addresses multiple tasks.…”

Section: Related Workmentioning

confidence: 98%

“…Therefore, human class is not considered in this work. We would like to point out that due to these challenges, many of the recent works [6], [52] only deals with one class in real world environment. Another potential future direction is to investigate the usefulness of Harmonic Convolutional Networks (HCN) [82] for our auditory tasks.…”

Section: Limitations and Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

Dai¹,

Vasudevan²,

Matas³

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 • camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of multiple vision 'teacher' methods and a sound 'student' method -the student method is trained to generate the same results as the teacher methods do. This way, the auditory system can be trained without using human annotations. To further boost the performance, we propose another novel auxiliary task, coined Spatial Sound Super-Resolution, to increase the directional resolution of sounds. We then formulate the four tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results show that 1) our method achieves good results for all four tasks, 2) the four tasks are mutually beneficial -training them together achieves the best performance, 3) the number and orientation of microphones are both important, and 4) features learned from the standard spectrogram and features obtained by the classic signal processing pipeline are complementary for auditory perception tasks. The data and code are released on the project page: https://www.trace.ethz.ch/publications/ 2020/sound perception/index.html.

show abstract

Section: Related Workmentioning

confidence: 98%

Section: Limitations and Future Workmentioning

confidence: 99%

Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

Dai¹,

Vasudevan²,

Matas³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This is because some classes are very rare in the middle-sized dataset, which already takes a great deal of effort to create. In comparison, the recent works [19,28] only deals with one class in real world environment. To our best knowledge, we are the first to work with multiple classes in an unconstrained real environment.…”

Section: Limitations and Future Workmentioning

confidence: 99%

“…This is because some classes are very rare in the middle-sized dataset. Due to this challenge, the recent works [19,29] only deals with one class in real world environment.…”

Section: Limitations and Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds

Vasudevan

Dai

Gool

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Humans can robustly recognize and localize objects by integrating visual and auditory cues. While machines are able to do the same now with images, less work has been done with sounds. This work develops an approach for dense semantic labelling of sound-making objects, purely based on binaural sounds. We propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 • camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of a vision 'teacher' method and a sound 'student' method -the student method is trained to generate the same results as the teacher method. This way, the auditory system can be trained without using human annotations. We also propose two auxiliary tasks namely, a) a novel task on Spatial Sound Super-resolution to increase the spatial resolution of sounds, and b) dense depth prediction of the scene. We then formulate the three tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results on the dataset show that 1) our method achieves good results for all the three tasks; and 2) the three tasks are mutually beneficial -training them together achieves the best performance and 3) the number and the orientations of microphones are both important. The data and code will be released on the project page.

show abstract