Speeding up Backpropagation

Silva, Fernando M.; Almeida, Luís B.

doi:10.1016/b978-0-444-88400-8.50022-4

Cited by 110 publications

(48 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We have already tested the adjustment of h during the training phase, but we did not achieved good results. We know that the variation of η during the training process improves the performance [9]. So, we plan to adjust η as a function of the error entropy instead of adjusting it as a function of the MSE.…”

Section: Discussionmentioning

confidence: 99%

Neural Network Classification Using Error Entropy Minimization

Santos

Alexandre²,

Sá³

2005

Biological and Artificial Intelligence Environments

View full text Add to dashboard Cite

Abstract. One way of using the entropy criteria in learning systems is to minimize the entropy of the error between two variables: typically, one is the output of the learning system and the other is the target. This framework has been used for regression. In this paper we show how to use the minimization of the entropy of the error for classification. The minimization of the entropy of the error implies a constant value for the errors. This, in general, does not imply that the value of the errors is zero. In regression, this problem is solved by making a shift of the final result such that it's average equals the average value of the desired target. We prove that, under mild conditions, this algorithm, when used in a classification problem, makes the error converge to zero and can thus be used in classification.

show abstract

Section: Discussionmentioning

confidence: 99%

Neural Network Classification Using Error Entropy Minimization

Santos

Alexandre²,

Sá³

2005

Biological and Artificial Intelligence Environments

View full text Add to dashboard Cite

show abstract

“…Unter diesen Verfahren sind die delta-delta-bzw. delta-bar-delta-Regel zu nennen (Jacobs, 1988;Silva und Almeida, 1990). Das bekannte Quickprop-Verfahren (Fahlmann, 1989) passt für die Bestimmung von η i eine nach oben geöffnete Parabel an die Fehlerfunktion an.…”

Section: Uberblicküber Andere Verbesserte Gd-verfahrenunclassified

Prognose makroökonomischer Zeitreihen: Ein Vergleich linearer Modelle mit neuronalen Netzen

Köller¹

2014

View full text Add to dashboard Cite

This thesis examines the potential of neural networks (NN) for forecasting of macroeconomic time series in comparison with linear models. The emphasis is on the autoregressive neural network (ARNN) model, which can be seen as a generalisation of the conventional autoregressive (AR) model where the non-linear part is implemented by a neural network of the feedforward type.Among the properties of macroeconomic time series that motivate various adaptations and enhancements both of linear and of neural network methodology are the following: high stochasticity, short length, non-stationary and seasonal behaviour and the fact that the existence of non-linearity is not clearly known a priori. The development and evaluation of the methods is based on simulated time series with known non-linear properties and on two macroeconomic time series: Austrian unemployment rate and Austrian industrial production index. Both series are in monthly observations and are seasonally unadjusted.After the introduction, the first main chapter of the thesis summarizes the basic concepts of linear univariate time series analysis and applies the linear methodology to the two selected macroeconomic time series. Particular attention is devoted to the problem of finding the appropriate differencing filter -conventional differences, seasonal differences or both -as a way to induce stationarity. Amongst other methods unit root tests are employed. The used linear models are the autoregressive (AR) and the autoregressive moving average (ARMA) model. These are augmented by a model part for deterministic seasonality. The models estimated on the two time series have rather high model order and, despite using a sparse modelling technique, comprise relatively many coefficients.The second main chapter treats various theoretic aspects and types of nonlinearity and carries out a sequence of hypothesis tests for non-linearity, in order to be able to rate the possible benefits of applying neural network methods to the two selected time series. The results of these tests indicate the presence of non-linear structure of the additive type, which can be exploited for better forecasting with ARNN models. However, these results have to be interpreted with care, as they might bei compromised by non-linearity of the multiplicative type, residual linear structure and possible structural breaks.The third main chapter introduces the ARNN model and develops methods for its estimation and specification. The specification contains a model part of determistic seasonality, includes a linear part and allows for sparse specification of coefficients. The generalisation capability of the ARNN model is to be ensured by four alternative model building approaches: the statistical-parametric approach with hypothesis testing and pruning, the classical approach with early stopping, the approach with regularisation and the Bayesian evidence framework. For each one of these approaches modifications and additional heuristics are contributed that seem appropriate in view of the a...

show abstract

“…Due to the integration of the system of (8) backward through time, this algorithm is sometimes called 'backpropagation through time'. In order to reduce the time of the learning process, the acceleration method of Silva & Almeida, (1990) was used, where each weight and time constant has its own adaptative learning rate. Cheron et al, 1996 IEEE) …”

Section: Methodological Adaptationsmentioning

confidence: 99%