Interpreting Intermediate Convolutional Layers In Unsupervised Acoustic Word Classification

Beguš, Gašper; Zhou, Alan

doi:10.1109/icassp43922.2022.9746849

Cited by 5 publications

(35 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Peak latency is an interpretable feature that can be analyzed with standard acoustic methods in deep convolutional networks. We analyze peak latency encoding with a technique for visualization of intermediate convolutional layers (in Beguš and Zhou 2021, 2022) that uses summation to identify peak activity in intermediate convolutional layers relative to the input/output. Because encoding of VOT duration in the form of peak latency appears to be present already at the brain stem level (based on the cABR experiment), we conduct the comparison of peak latency on the immediately preceding — second-to-final — convolutional layer relative to the input/output and parallel this information to the cABR signal in the brain stem relative to the stimulus.…”

Section: Discussionmentioning

confidence: 99%

“…The following properties have been shown to be robustly encoded in the second to last convolutional layer: periodicity and F0 together with F0 transitions, low frequency formant structure (F1 and to lesser degree F2), burst, and timing of individual segments (Beguš and Zhou, 2021, 2022). Figures 1a,b,c,d; 3a,b,c,d; and 4a,b,c,d,e illustrate the similarities between the signal from intermediate convolutional layers (obtained by the proposed technique in Beguš and Zhou 2021, 2022) and the cABR signal. Later convolutional layers do not encode all these acoustic properties (Beguš and Zhou, 2021, 2022).…”

Section: Discussionmentioning

confidence: 99%

“…The visualization technique in Beguš and Zhou (2021, 2022) allows us to test acoustic representations of intermediate convolutional layers of both the Generator that mimics the production principle in speech and the Discriminator that mimics the perception principle. Here, we compare the outputs of the proposed technique to outputs of brain imaging experiments.…”

Section: Methodsmentioning

confidence: 99%

“…WaveGAN itself does not contain any visualization techniques. For analyzing and visualization of intermediate convolutional layers, we use a technique proposed in Beguš and Zhou (2021) (for the Generator) and Beguš and Zhou (2022) for the classifier network (which has almost identical structure to the Discriminator). Beguš and Zhou (2021, 2022) argue that averaging over feature maps after ReLU activation yields a highly interpretable time series data for each convolutional layer that summarize what acoustic properties are encoded at which layer.…”

Section: Computational Experimentsmentioning

confidence: 99%

“…Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. Using a technique proposed by Beguš and Zhou (2022), we can analyze encoding of any acoustic property in intermediate convolutional layers of an artificial neural network. The proposed technique is based on a similar principle that underlies EEG: averaging of neural (artificial or biological) activity across neurons in the time domain.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Encoding of speech in convolutional layers and the brain stem based on language experience

Beguš¹,

Zhou²,

Zhao

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Comparing artificial neural networks (ANNs) with outputs of brain imaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. Using a technique proposed by Begus and Zhou (2021b), we can analyze encoding of any acoustic property in intermediate convolutional layers of an artificial neural network. This allows us to test similarities in speech encoding between the brain and artificial neural networks in a way that is more interpretable than the majority of existing proposals that focus on correlations and supervised models. We introduce fully unsupervised deep generative models (the Generative Adversarial Network architecture) trained on raw speech to the brain-and-ANN-comparison paradigm, which enable testing of both the production and perception principles in human speech. We present a framework that parallels electrophysiological experiments measuring complex Auditory Brainstem Response (cABR) in human brain with intermediate layers in deep convolutional networks. We compared peak latency in cABR relative to the stimulus in the brain stem experiment, and in intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examined and compared the effect of prior language exposure on the peak latency in cABR, and in intermediate convolutional layers of a phonetic property. Specifically, the phonetic property (i.e., VOT =10 ms) is perceived differently by English vs. Spanish speakers as voiced (e.g. [ba]) vs voiceless (e.g. [pa]). Critically, the cABR peak latency to the VOT phonetic property is different between English and Spanish speakers, and peak latency in intermediate convolutional layers is different between English-trained and Spanish-trained computational models. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Computational Experimentsmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Encoding of speech in convolutional layers and the brain stem based on language experience

Beguš¹,

Zhou²,

Zhao

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Encoding of speech in convolutional layers and the brain stem based on language experience

Beguš

Zhou

Zhao

2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

Comparing artificial neural networks with outputs of neuroimaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. The proposed technique is based on a similar principle that underlies electroencephalography (EEG): averaging of neural (artificial or biological) activity across neurons in the time domain, and allows to compare encoding of any acoustic property in the brain and in intermediate convolutional layers of an artificial neural network. Our approach allows a direct comparison of responses to a phonetic property in the brain and in deep neural networks that requires no linear transformations between the signals. We argue that the brain stem response (cABR) and the response in intermediate convolutional layers to the exact same stimulus are highly similar without applying any transformations, and we quantify this observation. The proposed technique not only reveals similarities, but also allows for analysis of the encoding of actual acoustic properties in the two signals: we compare peak latency (i) in cABR relative to the stimulus in the brain stem and in (ii) intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examine and compare the effect of prior language exposure on the peak latency in cABR and in intermediate convolutional layers. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property and for other neuroimaging techniques.

show abstract

Interpreting Intermediate Convolutional Layers of Generative CNNs Trained on Waveforms

Beguš

Zhou

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

This paper presents a technique to interpret and visualize intermediate layers in generative CNNs trained on raw speech data in an unsupervised manner. We argue that averaging over feature maps after ReLU activation in each transpose convolutional layer yields interpretable time-series data. This technique allows for acoustic analysis of intermediate layers that parallels the acoustic analysis of human speech data: we can extract F0, intensity, duration, formants, and other acoustic properties from intermediate layers in order to test where and how CNNs encode various types of information. We further combine this technique with linear interpolation of a model's latent space to show a causal relationship between individual variables in the latent space and activations in a model's intermediate convolutional layers. In particular, observing the causal effect between linear interpolation and the resulting changes in intermediate layers can reveal how individual latent variables get transformed into spikes in activation in intermediate layers.We train and probe internal representations of two modelsa bare WaveGAN architecture and a ciwGAN extension which forces the Generator to output informative data and results in the emergence of linguistically meaningful representations. Interpretation and visualization is performed for three basic acoustic properties of speech: periodic vibration (corresponding to vowels), aperiodic noise vibration (corresponding to fricatives), and silence (corresponding to stops). The proposal also allows testing of higher-level morphophonological alternations such as reduplication (copying). In short, using the proposed technique, we can analyze how linguistically meaningful units in speech get encoded in each convolutional layer of a generative neural network.

show abstract

Interpreting Intermediate Convolutional Layers In Unsupervised Acoustic Word Classification

Cited by 5 publications

References 18 publications

Encoding of speech in convolutional layers and the brain stem based on language experience

Encoding of speech in convolutional layers and the brain stem based on language experience

Encoding of speech in convolutional layers and the brain stem based on language experience

Interpreting Intermediate Convolutional Layers of Generative CNNs Trained on Waveforms

Contact Info

Product

Resources

About