Deep neural network models of sound localization reveal how perception is adapted to real-world environments

Francl, Andrew; McDermott, Josh H.

doi:10.1038/s41562-021-01244-z

Cited by 42 publications

(58 citation statements)

References 141 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These neurograms have been used for training ASRs to replicate human behavioral results. ASRs trained on neurograms from acoustic-hearing models have accurately predicted normal-hearing behavioral results for closed-set word recognition in noise [17], [18], pitch perception [19], and sound localization [20]. ASRs trained on CI outputs or neurograms from electric-stimulation models have predicted closed-set word recognition rates in noise for CI listeners and have provided some insights into factors that may underlie variability in CI outcomes, such as current spread, neural survival, and cognitive noise [21]- [23].…”

Section: Introductionmentioning

confidence: 95%

From Microphone to Phoneme: An End-to-End Computational Neural Model for Predicting Speech Perception With Cochlear Implants

Brochier

Schlittenlacher

Roberts

et al. 2022

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

Goal: Advances in computational models of biological systems and artificial neural networks enable rapid virtual prototyping of neuroprostheses, accelerating innovation in the field. Here, we present an end-to-end computational model for predicting speech perception with cochlear implants (CI), the most widely-used neuroprosthesis. Methods: The model integrates CI signal processing, a finite element model of the electrically-stimulated cochlea, and an auditory nerve model to predict neural responses to speech stimuli. An automatic speech recognition neural network is then used to extract phoneme-level speech perception from these neural response patterns. Results: Compared to human CI listener data, the model predicts similar patterns of speech perception and misperception, captures between-phoneme differences in perceptibility, and replicates effects of stimulation parameters and noise on speech recognition. Information transmission analysis at different stages along the CI processing chain indicates that the bottleneck of information flow occurs at the electrode-neural interface, corroborating studies in CI listeners. Conclusion: An end-to-end model of CI speech perception replicated phoneme-level CI speech perception patterns, and was used to quantify information degradation through the CI processing chain. Significance: This type of model shows great promise for developing and optimizing new and existing neuroprostheses.

show abstract

Section: Introductionmentioning

confidence: 95%

From Microphone to Phoneme: An End-to-End Computational Neural Model for Predicting Speech Perception With Cochlear Implants

Brochier

Schlittenlacher

Roberts

et al. 2022

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

show abstract

“…The discrepancies shown here for model metamers contrast with a growing number of examples of compelling human-model similarities for behavioral judgments of natural stimuli. Models optimized for object recognition (90), speech recognition (5), sound localization (91), and pitch recognition (92) all exhibit qualitative and often quantitative similarities to human judgments when run in traditional psychophysical experiments with natural or relatively naturalistic stimuli. These results suggest that neural network models trained in naturalistic conditions often match human behavior for signals that fall within their training distribution, but not for some signals derived from the model that fall outside the distribution of natural sounds and images.…”

Section: Future Directionsmentioning

confidence: 99%

Model metamers illuminate divergences between biological and artificial neural networks

Feather

Leclerc

Mądry

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural network models of sensory systems are often proposed to learn representational transformations with invariances like those in the brain. To reveal these invariances we generated “model metamers” – stimuli whose activations within a model stage are matched to those of a natural stimulus. Metamers for state-of-the-art supervised and unsupervised neural network models of vision and audition were often completely unrecognizable to humans when generated from deep model stages, suggesting differences between model and human invariances. Targeted model changes improved human-recognizability of model metamers, but did not eliminate the overall human-model discrepancy. The human-recognizability of a model’s metamers was well predicted by their recognizability by other models, suggesting that models learn idiosyncratic invariances in addition to those required by the task. Metamer recognition dissociated from both traditional brain-based benchmarks and adversarial vulnerability, revealing a distinct failure mode of existing sensory models and providing a complementary benchmark for model assessment.

show abstract

“…To reduce possible biases by a specific architecture, unless otherwise stated, the reported recognition accuracy, TMTFs, and neurophysiological similarities are averages of the results of the four models with the highest recognition accuracies. This could be considered as a modeled version of reporting average quantities in multiple participants in human studies (Francl and McDermott, 2022). Four architectures with 13 layers were selected by performing an architecture search (see the methods for the detailed procedure).…”

Section: Optimizing a Neural Network For Natural Sound Recognitionmentioning

confidence: 99%

“…Note that we did not directly model or try to reproduce human AM sensitivity in the optimization process. This kind of two-step optimization-and-analysis procedure has explained a number of properties of the auditory system, such as cochlear frequency tuning (Lewicki, 2002), AM tuning (Khatami and Escabí, 2020), the receptive field in the auditory cortex (Terashima and Okada, 2012), pitch perception (Saddler et al, 2021), sound localization (Francl and McDermott, 2022), and speech processing (Ashihara et al, 2021), as well as in other sensory modalities (Kriegeskorte and Douglas, 2018). To reduce the number of hard-coded assumptions and clarify the relationship between the optimization procedure and the emergent properties, we applied an NN directly to a “raw” sound waveform without any preprocessing (Hoshen et al, 2015; Tokozume and Harada, 2017).…”

Section: Introductionmentioning

confidence: 99%

Human-like Modulation Sensitivity Emerging through Optimization to Natural Sound Recognition

Koumura¹,

Terashima²,

Furukawa³

2022

Preprint

View full text Add to dashboard Cite

Natural sounds contain rich patterns of amplitude modulation (AM), which is one of the essential sound dimensions for hearing perception. The sensitivity of human hearing to AM measured by psychophysics takes diverse forms, from low-pass to high-pass, depending on the experimental conditions. Here, we address with a single framework the questions of why such patterns of AM sensitivity have emerged in the human auditory system and how they are realized by our neural mechanisms. Assuming that optimization for natural sound recognition has taken place during human evolution and development, we examined its effect on the formation of AM sensitivity by optimizing a computational model, specifically, a multi-layer (or deep) neural network, for natural sound recognition and simulating psychophysical experiments in which the model's AM sensitivity was measured. Relatively higher layers in the optimized model exhibited qualitatively and quantitatively similar AM sensitivity to that of humans, even though the model was not designed to reproduce human-like AM sensitivity. The similarity of the model's AM sensitivity to humans' correlated with its sound recognition accuracy. Optimization of the model to degraded sounds revealed the necessity of natural AM patterns for the emergence of human-like AM sensitivity. Consistent results were observed from optimizations to two different types of natural sound. Moreover, simulated neurophysiological experiments on the same model revealed a correspondence between the model layers and the auditory brain regions that is based on the similarity of their neural AM tunings. The layers in which human-like psychophysical AM sensitivity emerged exhibited substantial neurophysiological similarity with the auditory midbrain and higher regions. These results suggest that the behavioral AM sensitivity of human hearing has emerged as a result of optimization for natural-sound recognition in the course of our evolution and/or development and that it is based on a stimulus representation encoded in the neural firing rates in the auditory midbrain and higher regions.

show abstract

Deep neural network models of sound localization reveal how perception is adapted to real-world environments

Cited by 42 publications

References 141 publications

From Microphone to Phoneme: An End-to-End Computational Neural Model for Predicting Speech Perception With Cochlear Implants

From Microphone to Phoneme: An End-to-End Computational Neural Model for Predicting Speech Perception With Cochlear Implants

Model metamers illuminate divergences between biological and artificial neural networks

Human-like Modulation Sensitivity Emerging through Optimization to Natural Sound Recognition

Contact Info

Product

Resources

About