Theano: new features and speed improvements

Bastien, Frédéric; Lamblin, Pascal; Pascanu, Razvan; Bergstra, James; Goodfellow, Ian; Bergeron, André; Bouchard, Nicolas; Warde-Farley, David; Bengio, Yoshua

doi:10.48550/arxiv.1211.5590

Cited by 81 publications

(105 citation statements)

References 7 publications

Supporting

Mentioning

105

Contrasting

Order By: Relevance

“…The best model during the entire run was kept. All experiments were carried out using Python and implemented in Theano [16,17].…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Kohan,

Rietman,

Siegelmann

2018

Preprint

View full text Add to dashboard Cite

We introduce Error Forward-Propagation, a biologically plausible mechanism to propagate error feedback forward through the network. Architectural constraints on connectivity are virtually eliminated for error feedback in the brain; systematic backward connectivity is not used or needed to deliver error feedback. Feedback as a means of assigning credit to neurons earlier in the forward pathway for their contribution to the final output is thought to be used in learning in the brain. How the brain solves the credit assignment problem is unclear. In machine learning, error backpropagation is a highly successful mechanism for credit assignment in deep multilayered networks. Backpropagation requires symmetric reciprocal connectivity for every neuron. From a biological perspective, there is no evidence of such an architectural constraint, which makes backpropagation implausible for learning in the brain. This architectural constraint is reduced with the use of random feedback weights. Models using random feedback weights require backward connectivity patterns for every neuron, but avoid symmetric weights and reciprocal connections. In this paper, we practically remove this architectural constraint, requiring only a backward loop connection for effective error feedback. We propose reusing the forward connections to deliver the error feedback by feeding the outputs into the input receiving layer. This mechanism, Error Forward-Propagation, is a plausible basis for how error feedback occurs deep in the brain independent of and yet in support of the functionality underlying intricate network architectures. We show experimentally that recurrent neural networks with two and three hidden layers can be trained using Error Forward-Propagation on the MNIST and Fashion MNIST datasets, achieving 1.90% and 11% generalization errors respectively.

show abstract

“…The best model during the entire run was kept. All experiments were carried out using Python and implemented in Theano [16,17].…”

Section: Resultsmentioning

confidence: 99%

“…More complex architectures are possible in our implementation, but are not considered here. The model was implemented in Python and Theano [16,17].…”

Section: Implementation Of the Modelmentioning

confidence: 99%

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Kohan,

Rietman,

Siegelmann

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…To calculate the momentum at a position in posterior space we need to calculate the derivative of the posterior density for the local environment. With the advances in automatic differentiation in tools like, for example, Theano [48], which is used in the pymc3 framework [49], it is possible to quickly calculate the derivatives of more complex functions.…”

Section: Sampling Algorithmmentioning

confidence: 99%

Uncertainty Introduced by Darkening Agents in the Lunar Regolith: An Unmixing Perspective

et al. 2021

View full text Add to dashboard Cite

On the Moon, in the near infrared wavelength range, spectral diagnostic features such as the 1-μm and 2-μm absorption bands can be used to estimate abundances of the constituent minerals. However, there are several factors that can darken the overall spectrum and dampen the absorption bands. Namely, (1) space weathering, (2) grain size, (3) porosity, and (4) mineral darkening agents such as ilmenite have similar effects on the measured spectrum. This makes spectral unmixing on the Moon a particularly challenging task. Here, we try to model the influence of space weathering and mineral darkening agents and infer the uncertainties introduced by these factors using a Markov Chain Monte Carlo method. Laboratory and synthetic mixtures can successfully be characterized by this approach. We find that the abundance of ilmenite, plagioclase, clino-pyroxenes and olivine cannot be inferred accurately without additional knowledge for very mature spectra. The Bayesian approach to spectral unmixing enables us to include prior knowledge in the problem without imposing hard constraints. Other data sources, such as gamma-ray spectroscopy, can contribute valuable information about the elemental abundances. We here find that setting a prior on TiO2 and Al2O3 can mitigate many of the uncertainties, but large uncertainties still remain for dark mature lunar spectra. This illustrates that spectral unmixing on the Moon is an ill posed problem and that probabilistic methods are important tools that provide information about the uncertainties, that, in turn, help to interpret the results and their reliability.

show abstract

“…Various platforms facilitating the construction and evaluation of deep neural networks exist. They include Tensorflow [1], PyTorch [55], Theano [2], and Caffe [22]. Using these platforms, derivatives used in gradients are computed automatically and hence the user can concentrate on the design and optimization of the network architecture and other aspects of learning.…”

Section: Deep Learningmentioning

confidence: 99%

Deep Node Ranking for Neuro-symbolic Structural Node Embedding and Classification

Škrlj,

Kralj,

Konc

et al. 2019

Preprint

View full text Add to dashboard Cite

Complex networks are used as an abstraction for systems modeling in physics, biology, sociology, and other areas. We propose an algorithm, named Deep Node Ranking (DNR), based on fast personalized node ranking and the approximation power of deep learning for learning supervised and unsupervised network embeddings, as well as for classifying network nodes directly. The experiments demonstrate that the DNR algorithm is competitive with strong baselines on nine node classification benchmarks from the domains of molecular biology, finance, social media and language processing in terms of speed, as well as predictive accuracy. Embeddings, obtained by the proposed algorithm, are also a viable option for network visualization.

show abstract

Theano: new features and speed improvements

Cited by 81 publications

References 7 publications

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning

Uncertainty Introduced by Darkening Agents in the Lunar Regolith: An Unmixing Perspective

Deep Node Ranking for Neuro-symbolic Structural Node Embedding and Classification

Contact Info

Product

Resources

About