An Incremental Framework Based on Cross-Validation for Estimating the Architecture of a Multilayer Perceptron

Abstract

We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using cross-validation. We consider five variants that implement forward/backward search, using single/multiple operators and searching depth-first/breadth-first. On 44 classification and 30 regression datasets, we exhaustively search for the optimal and evaluate the goodness based on: (1) Order, the accuracy with respect to the optimal and (2) Rank, the computational complexity. We check for the effect of two resampling methods (5 × 2, ten-fold cv), four statistical tests (5 × 2 cv t, ten-fold cv t, Wilcoxon, sign) and two corrections for multiple comparisons (Bonferroni, Holm). We also compare with Dynamic Node Creation (DNC) and Cascade Correlation (CC). Our results show that: (1) On most datasets, networks with few hidden units are optimal, (2) forward searching finds simpler architectures, (3) variants using single node additions (deletions) generally stop early and get stuck in simple (complex) networks, (4) choosing the best of multiple operators finds networks closer to the optimal, (5) MOST variants generally find simpler networks having lower or comparable error rates than DNC and CC.

A deterministic model selection scheme for incremental RBFNN construction in time series forecasting

“…Two model selection approaches based on a K-fold crossvalidation (K-CV) model evaluation methodology, which is commonly used in the literature [14,[24][25][26], are also applied for comparison purposes. The main differences with respect to MoSe, MoSe-R and MoSe-D model selection methodologies are in terms of the generation of the training and validation sets, the RBFNN optimization process and the iterative procedure typical of a cross-validation approach to evaluate a given model/structure network (Fig.…”

“…A network smaller than the optimal architecture underfits and fails to learn the data well (bias is high and variance is low), and a large network suffers the overfitting problem, resulting in poor generalization (bias is low and variance is high). Thus, the optimal architecture is the one with low bias and low variance so that the network learns the function underlying the data and not the existing noise [14].…”

“…Regardless of the approach chosen, when the network structure changes, an evaluation is needed to check the suitability of the new network [14]. For this purpose, there are information theory-based model evaluation methodologies such as the Akaike's information criterion (AIC) [21], the Bayesian information criterion (BIC) [22] and the minimum description length (MDL) [23].…”

“…-K-fold cross-validation: it is widely used in the literature [14,[24][25][26], but the value of K is based on a subjective judgment: a low value results in small variance and large bias, and a high value makes the evaluation computationally expensive [27]. -Leave-One-Out (LOO): it is less biased, but its variance [24] and computational cost are unacceptable.…”

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

“…The other common approaches for optimizing neural network architecture are basically growing, pruning and a combination of two strategies namely growing and pruning [13]. The first, also called as constructive methods, start with a minimal network and add new hidden units during the training process [14][15][16].…”