How Transferable Are Features in Convolutional Neural Network Acoustic Models across Languages?

Thompson, Jessica A. F.; Schönwiesner, Marc; Bengio, Yoshua; Willett, Daniel

doi:10.1109/icassp.2019.8683043

Cited by 17 publications

(14 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The freeze-trained transfer networks from Thompson et al (2019), which were initialized with parameters from a network previously trained on one language and then freeze-trained on another, outperformed all other freeze-trained networks (no transfer) and other transfer networks (no freeze training). Here, we compare the activations of the English standard, Dutch standard and Dutch-to-English freeze-trained networks from Thompson et al (2019). We predict with high confidence that the early layers of the Dutchto-English freeze-trained network will be more similar to the Dutch than the English standard model since they were initialized with the parameters from the Dutch standard network and received relatively little training afterwards.…”

Section: Freeze Trainingmentioning

confidence: 97%

The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis

Thompson

Bengio

Schoenwiesner

2019

2019 Conference on Cognitive Computational Neuroscience

Self Cite

View full text Add to dashboard Cite

Centered Kernel Alignment (CKA) was recently proposed as a similarity metric for comparing activation patterns in deep networks. Here we experiment with the modified RV-coefficient (RV2), which has similar properties to CKA while being less sensitive to dataset size. We compare the representations of networks that received varying amounts of training on different layers: a standard trained network (all parameters updated at every step), a freeze-trained network (layers gradually frozen during training), random networks (only some layers trained), and a completely untrained network. We found that RV2 was able to recover expected similarity patterns and provide interpretable similarity matrices that suggested hypotheses about how representations are affected by different training recipes. We propose that the superior performance achieved by freeze training can be attributed to representational differences in the penultimate layer. Comparisons to random networks suggest that the inputs and targets serve as anchors on the representations in the lowest and highest layers.

show abstract

Section: Freeze Trainingmentioning

confidence: 97%

The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis

Thompson

Bengio

Schoenwiesner

2019

2019 Conference on Cognitive Computational Neuroscience

Self Cite

View full text Add to dashboard Cite

show abstract

“…Freeze training refers to the procedure by which layers are gradually removed from the set of trainable variables over the course of training and in order of depth. Previous work has shown that freeze training can speed up training (Raghu et al, 2017) and facilitate transfer across related tasks (Thompson et al, 2019a).…”

Section: Convolutional Neural Network Activationsmentioning

confidence: 99%

“…This quilting procedure, described below, allowed us to focus our comparison on representational transformations only up to the sub-word level in both the convnets and the human auditory system. The audio corpora from which the stimuli were constructed were the same datasets that were used in (Thompson et al, 2019a) and (Thompson et al, 2019b), which are owned by Nuance Communications. Each of the three datasets, one for English, Dutch and German, contained 64-83 hours of spoken text read by several native speakers in a quiet room.…”

Section: Experimental Stimulimentioning

confidence: 99%

See 1 more Smart Citation

Training neural networks to recognize speech increased their correspondence to the human auditory pathway but did not yield a shared hierarchy of acoustic features

Thompson

Bengio

Formisano

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The correspondence between the activity of artificial neurons in convolutional neural networks (CNNs) trained to recognize objects in images and neural activity collected throughout the primate visual system has been well documented. Shallower layers of CNNs are typically more similar to early visual areas and deeper layers tend to be more similar to later visual areas, providing evidence for a shared representational hierarchy. This phenomenon has not been thoroughly studied in the auditory domain. Here, we compared the representations of CNNs trained to recognize speech (triphone recognition) to 7-Tesla fMRI activity collected throughout the human auditory pathway, including subcortical and cortical regions, while participants listened to speech. We found no evidence for a shared representational hierarchy of acoustic speech features. Instead, all auditory regions of interest were most similar to a single layer of the CNNs: the first fully-connected layer. This layer sits at the boundary between the relatively task-general intermediate layers and the highly task-specific final layers. This suggests that alternative architectural designs and/or training objectives may be needed to achieve fine-grained layer-wise correspondence with the human auditory pathway.HighlightsTrained CNNs more similar to auditory fMRI activity than untrainedNo evidence of a shared representational hierarchy for acoustic featuresAll ROIs were most similar to the first fully-connected layerCNN performance on speech recognition task positively associated with fmri similarity

show abstract

“…As an example of other machine learning models applied to EBSD, a nearest neighbor machine learning model has been previously explored by Goulden et al (2017) to address the challenge of separating ferrite and martensite by machine learning aided pattern quality analysis, but pattern quality is too rigid a metric for general use, and the method was reliant on human analysis and confirmation over multiple rounds of indexing a single map. Another advantage is the flexibility of these CNNs, allowing for the transfer of knowledge learned from discriminating images in other contexts (Gonzalez et al, 2019;Thompson et al, 2019), the development of models suitable for application in a highly specific materials space, or deployment in an application where the phases present are completely unknown. For example, during the initial analysis of a material, a pre-trained CNN could be utilized for the determination of which Bravais lattices or space groups are present (Kaufmann et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Phase Mapping in EBSD Using Convolutional Neural Networks

et al. 2020

View full text Add to dashboard Cite

The emergence of commercial electron backscatter diffraction (EBSD) equipment ushered in an era of information rich maps produced by determining the orientation of user-selected crystal structures. Since then, a technological revolution has occurred in the quality, rate detection, and analysis of these diffractions patterns. The next revolution in EBSD is the ability to directly utilize the information rich diffraction patterns in a high-throughput manner. Aided by machine learning techniques, this new methodology is, as demonstrated herein, capable of accurately separating phases in a material by crystal symmetry, chemistry, and even lattice parameters with fewer human decisions. This work is the first demonstration of such capabilities and addresses many of the major challenges faced in modern EBSD. Diffraction patterns are collected from a variety of samples, and a convolutional neural network, a type of machine learning algorithm, is trained to autonomously recognize the subtle differences in the diffraction patterns and output phase maps of the material. This study offers a path to machine learning coupled phase mapping as databases of EBSD patterns encompass an increasing number of the possible space groups, chemistry changes, and lattice parameter variations.

show abstract

How Transferable Are Features in Convolutional Neural Network Acoustic Models across Languages?

Cited by 17 publications

References 5 publications

The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis

The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis

Training neural networks to recognize speech increased their correspondence to the human auditory pathway but did not yield a shared hierarchy of acoustic features

Phase Mapping in EBSD Using Convolutional Neural Networks

Contact Info

Product

Resources

About