Nonlinear principal component analysis by neural networks

Abstract

Abstract-With very noisy data, overfitting is a serious problem in pattern recognition. For nonlinear regression, having plentiful data eliminates overfitting, but for nonlinear principal component analysis (NLPCA), overfitting persists even with plentiful data. Thus simply minimizing mean square error is not a sufficient criterion for NLPCA to find good solutions in noisy data.A new index is proposed which measures the disparity between the nonlinear principal components u andũ for a data point x and its nearest neighbourx. This index, 1 − CS (the Spearman rank correlation between u andũ), tends to increase with overfitted solutions, thereby providing a diagnostic tool to determine how much regularization (i.e. weight penalty) should be used in the objective function of the NLPCA to prevent overfitting. Tests are performed using autoassociative neural networks for NLPCA on synthetic and real climate data. I. INTRODUCTIONIn principal component analysis (PCA), a given dataset is approximated by a straight line, which minimizes the mean square error (MSE) -pictorially, in a scatterplot of the data, the straight line found by PCA passes through the 'middle' of the dataset. In nonlinear PCA (NLPCA), the straight line in PCA is replaced by a curve. NLPCA can be performed by a variety of methods, e.g. the autoassociative neural network (NN) model [6,5], and the kernel PCA model [11].When using nonlinear machine learning methods, the presence of noise in the data can lead to overfitting (i.e. fitting to the noise). When plentiful data are available (i.e. far more samples than model parameters), overfitting is not a problem when performing nonlinear regression on noisy data. Unfortunately, even with plentiful data, overfitting is a problem when applying NLPCA to noisy data [4,2]. As illustrated in Figure 1, overfitting in NLPCA can arise from the geometry of the problem, rather than from the scarsity of data. Here for a Gaussian-distributed data cloud, a nonlinear model with enough flexibility will find the zigzag solution of Figure 1b as having a smaller MSE than the linear solution in Figure 1a. Since the distance between the point A and a, its projection on the NLPCA curve, is smaller in Figure 1b than the corresponding distance in Figure 1a, it is easy to see that the more zigzags there are in the curve, the smaller is the MSE. However, the two neighbouring points A and B, on opposite sides of an "ambiguity" line [8], are projected far apart on the NLPCA curve in Figure 1b. Thus simply searching for the solution which gives the smallest MSE is not a sufficient criterion for NLPCA to find a satisfactory solution in a highly noisy dataset.Regularization (e.g. the addition of weight penalty or decay terms in the objective functions in NN models) has been

Sea surface temperature patterns in the Tropical Atlantic: Principal component analysis and nonlinear principal component analysis

“…There is an important difference between PCA and rotated PCA methods; as it is generally impossible to have a simultaneous solution that explains the maximum global variance of the data and approaching pattern recognition. NLPCA can give both types of information, thus the nonlinearity in NLPCA unifies the PCA and rotated PCA approaches (Hsieh 2001). In this paper, in terms of variance, the first rotated PCA explained 31% of the total variance (not shown), versus 36% using the first PCA mode.…”

“…That the classical PCA is indeed a linear version of this NLPCA can be readily seen by replacing all of the transfer functions with the identity function, thereby removing the NLPCA nonlinear modeling capability (Hsieh 2001). The forward map to u then involve only a linear combination of the original variables as in the PCA.…”

“…8 and 9). The NLPCA network architectures used for calculating NLPCA SST mode 2 are essentially the same as those of Hsieh (2001); three hidden layer neurons with all training and fitting parameters identical to the configuration used for calculating NLPCA SST mode 1. The explained variance results show that the NLPCA mode 2 explains more variance as the second principal components.…”

“…The first 20 PCs are then used as the input to the NLPCA network. Each input variable is normalized by subtracting its mean and then dividing by the standard deviation of the first PC (Hsieh 2001). Scatter plots of the 3 leading principal component time series are shown by the solid blue dots in Fig.…”

“…It has been applied on SST to study climate variations in the tropical Pacific (e.g., Hsieh 2001;Li et al 2005). In particular, it has shown that the spatial variability associated with the ENSO phenomenon is non-linear (Hsieh 2004(Hsieh , 2007.…”