Healing Pains of the Past Using Neuronal Transplantation

Zheng, Xiaoting; Gandhi, Sunil

doi:10.1016/j.neuron.2016.12.012

Cited by 2 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As in Lee et al (2014), Szegedy et al (2015), and Wang et al (2015), we utilize auxiliary error functions to train internal layers directly in conjunction with the error from the output layer found via backpropagation. We propose an error function that maximizes the similarity between the representational spaces of a student DNN and that of a teacher model.…”

Section: Methodsmentioning

confidence: 99%

“…These terms are combined using weighting hyper parameter α, similar to the method discussed in Lee et al (2014), Szegedy et al (2015), and Wang et al (2015). In RDL, α is the weight of the RDL error in the overall error function.…”

Section: Methodsmentioning

confidence: 99%

“…Lee et al (2014) also showed that auxiliary error functions improve DNN representational learning. Instead of using semi-supervised methods, they performed classification with a softmax or L2SVM readout at a given intermediate hidden layer.…”

Section: Introductionmentioning

confidence: 99%

“…In this DNN, two auxiliary networks were used to directly backpropagate from two intermediate layers back through the main network. Similar to the method used in Lee et al (2014), the parameters for the layers in the main network directly connected to auxiliary networks were updated using a linear combination of the backpropagated gradients from later layers and the auxiliary network.…”

Section: Introductionmentioning

confidence: 99%

“…These networks consisted of a convolutional layer followed by three fully connected layers and a softmax classifier. As in Lee et al (2014) and Szegedy et al (2015), the auxiliary gradients were linearly combined to update the model parameters. Adding these supervised auxiliary error functions led to an improved accuracy for two very large datasets, ILSVRC12 (Russakovsky et al, 2014) and MIT Places (Zhou et al, 2014).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Representational Distance Learning for Deep Neural Networks

McClure

Kriegeskorte

2016

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Deep neural networks (DNNs) provide useful models of visual representational transformations. We present a method that enables a DNN (student) to learn from the internal representational spaces of a reference model (teacher), which could be another DNN or, in the future, a biological brain. Representational spaces of the student and the teacher are characterized by representational distance matrices (RDMs). We propose representational distance learning (RDL), a stochastic gradient descent method that drives the RDMs of the student to approximate the RDMs of the teacher. We demonstrate that RDL is competitive with other transfer learning techniques for two publicly available benchmark computer vision datasets (MNIST and CIFAR-100), while allowing for architectural differences between student and teacher. By pulling the student's RDMs toward those of the teacher, RDL significantly improved visual classification performance when compared to baseline networks that did not use transfer learning. In the future, RDL may enable combined supervised training of deep neural networks using task constraints (e.g., images and category labels) and constraints from brain-activity measurements, so as to build models that replicate the internal representational spaces of biological brains.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%