Quadratic neural unit is a good compromise between linear models and neural networks for industrial applications

Learning Entropy: Multiscale Measure for Incremental Learning

“…However, the design of a proper learning model and its correct pre-training are crucial and non-trivial tasks for correct evaluation of LE and it may require an expert in adaptive (learning) systems or in neural networks. Nevertheless, from our experiments with AP and HONU [1,2,[36][37][38][39][40] it appears that the very precise pre-training of the learning model is practically not always too crucial and that the structure of a learning model can be designed quite universally, e.g., with HONUs as they are nonlinear mapping predictors that are naturally linear in parameters. A practical rule of thumb for the above introduced HONU and GD can be to keep pre-training as long as the error criteria keep decreasing, i.e., until the learning model learns what it can in respect to its quality of approximation vs. the complexity of the data.…”

“…Moreover, GD learning is very efficient especially when used with linear filters or low-dimensional neural network architectures (predictors). The use of GD is recalled particularly for linear predictors (filters) and for polynomial predictors (also called Higher-Order Neural Units HONUs [1,[35][36][37]) in this subsection.…”

“…As a nonlinear and low-dimensional predictive model, static quadratic neural unit (QNU, [1,35,36]) is chosen for its good quality of nonlinear approximation and in-parameter linearity that theoretically avoids problem of local minima for adaptation [37]. QNU can be expressed in a long-vector multiplication form as follows:…”

“…This novel entropy concept can utilize sample-by-sample adaptation of low-dimensional predictors [1,2,[36][37][38][39] and uses the technique of the AP. Notice, the GD with normalized learning rate (e.g., [41,42]) is used in this paper for its clarity and for its good performance; however, LE is not principally limited to only GD technique, nor to supervised techniques in general.…”

“…For evaluation of LE of nonlinear time series, polynomial adaptive predictors such as Higher-Order Neural Units can be recommended [1,[35][36][37]; HONUs are attractive adaptive models because their mapping is customable as nonlinear while they are linear in parameters => optimization of HONUs is of a linear nature, so HONUs do not suffer from local minima problem in the way as conventional neural networks do when GD learning technique is used.…”