Deep neural network models of sound localization reveal how perception is adapted to real-world environments

Abstract

Mammals localize sounds using information from their two ears. Localization in real-world conditions is challenging, as echoes provide erroneous information, and noises mask parts of target sounds. To better understand real-world localization we equipped a deep neural network with human ears and trained it to localize sounds in a virtual environment. The resulting model localized accurately in realistic conditions with noise and reverberation, outperforming alternative systems that lacked human ears. In simulated experiments, the network exhibited many features of human spatial hearing: sensitivity to monaural spectral cues and interaural time and level differences, integration across frequency, and biases for sound onsets. But when trained in unnatural environments without either reverberation, noise, or natural sounds, these performance characteristics deviated from those of humans. The results show how biological hearing is adapted to the challenges of real-world environments and illustrate how artificial neural networks can extend traditional ideal observer models to real-world domains.

Deep neural network models reveal interplay of peripheral coding and stimulus statistics in pitch perception

“…DNNs provide general-purpose architectures that can be optimized to perform challenging real-world tasks 13 . While DNNs are unlikely to fully achieve optimal performance, they might reveal the effects of optimizing a system under particular constraints 14,15 . Previous work has documented similarities between human and network behavior for neural networks trained on vision or hearing tasks [16][17][18] .…”

Inferring the basis of binaural detection with a modified autoencoder

“…Here, we set out to discover the efficacy of DNNs as a means to infer proximate mechanisms of audition underlying a targeted auditory phenomena. We modelled binaural hearing (namely, in the context of signal detection), representative of a highly specialized system for which the computations within DNN analogues have barely been examined [11][12][13]. We found that a relatively shallow DNN was able to successfully utilize binaural discrepancies in auditory stimuli, as opposed to seeking an alternative strategy [29] and/or failing to exhibit binaural detection behavior.…”

“…Further, binaural detection is a highly specialised auditory function for which deficits have real-world consequences 25,26 . DNNs may offer the opportunity to bridge this gap between animal and human data, and as yet, the inner workings of DNNs constructed to handle binaural audio have scarcely been considered [27][28][29] .…”

“…We know that specialized binaural mechanisms are able to utilize interaural time differences (ITDs) less than one ten thousandth of a second to improve the detection of a signal in noise [9,10]. Yet, to date, the inner workings of DNN analogues configured to handle binaural audio have scarcely been considered [11][12][13]. Here, we trained a DNN, with a modified autoencoder architecture, to imitate the phenomena of binaural signal detection.…”

Deep learning in alternate reality

“…A body of recent research has pointed out similarities and differences between such deep neural networks and the primate visual system [1,2], but comparable auditory studies remained scarce [3,4]. A new article by Francl and McDermott [5] fills this gap, by revealing how properties of human audition can emerge in deep networks, and how such properties are causally dependent on features of the natural environment.…”