Regularizing Deep Neural Networks by Enhancing Diversity in Feature Extraction

Ayinde, Babajide O.; Inanc, Tamer; Żurada, Jacek M.

doi:10.1109/tnnls.2018.2885972

Cited by 65 publications

(30 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, deep learning algorithms have made a series of revolutions in the field of machine learning (Huang et al 2019) since the classification capability of a neural network to fit a decision boundary plane has become significantly more reliable (LeCun et al 2015) which can successfully learn and extract patterns and unique features from big data (Ayinde et al 2019). Deep learning also can effectively avoid local optimization and eliminates the need to set model parameters because of autonomous processes (Zhang et al 2017).…”

Section: Introductionmentioning

confidence: 99%

An extreme rainfall-induced landslide susceptibility assessment using autoencoder combined with random forest in Shimane Prefecture, Japan

Nam

Wang

2020

Geoenviron Disasters

View full text Add to dashboard Cite

Background: Landslide-affecting factors are uncorrelated or non-linearly correlated, limiting the predictive performance of traditional machine learning methods for landslide susceptibility assessment. Deep learning methods can take advantage of the high-level representation and reconstruction of information from landslide-affecting factors. In this paper, a novel deep learning-based algorithm that combine classifiers of both deep learning and machine learning is proposed for landslide susceptibility assessment. A stacked autoencoder (StAE) and a sparse autoencoder (SpAE) both consist of an input layer for raw data, hidden layer for feature extraction, and output layer for classification and prediction. As a study case, Oda City and Gotsu City in Shimane Prefecture, southwestern Japan, were used for susceptibility assessment and prediction of landslides triggered by extreme rainfall. Results: The prediction performance was compared by analyzing real landslide and non-landslide data. The prediction performance of random forest (RF) was evaluated as better than that of a support vector machine (SVM) in traditional machine learning, so RF was combined with both StAE and SpAE. The results show that the prediction ratio of the combined classifiers was 93.2% for StAE combined with RF model and 92.5% for SpAE combined with RF model, which were higher than those of the SVM (87.4%), RF (89.7%), StAE (84.2%), and SpAE (88.2%). Conclusions: This study provides an example of combined classifiers giving a better predictive ratio than a single classifier. The asymmetric and unsupervised autoencoder combined with RF can exploit optimal non-linear features from landslide-affecting factors successfully, outperforms some conventional machine learning methods, and is promising for landslide susceptibility assessment.

show abstract

Section: Introductionmentioning

confidence: 99%

An extreme rainfall-induced landslide susceptibility assessment using autoencoder combined with random forest in Shimane Prefecture, Japan

Nam

Wang

2020

Geoenviron Disasters

View full text Add to dashboard Cite

show abstract

“…We should note that the dictionary learning in sparse coding (1) and AE (4) differ by two aspects. Firstly, the reconstruction error (4) involves mapping of data into itself by two matrices W 1 and W 2 , while the same error being the first term of (1) involves one matrix Φ. Secondly, (1) is solved by optimization, while (4) is based on unsupervised learning of h.…”

Section: Dictionary Learning Via Constrained Autoencodersmentioning

confidence: 99%

“…41 shows the distribution of pairwise correlation of first hidden layer features ( Test error(%) on MNIST. Source: [4] As mentioned earlier, a crucial step in achieving good performance with divReg-2 is not only in the choice of τ * but also in the initialization of adaptive dropout fraction α. Figs.…”

Section: Feature Evolution During Trainingmentioning

confidence: 99%

“…This respectively. Source: [4] ence (SNLI) dataset [149]. The original dataset contains 550, 152 duos of premisehypothesis sentences and their corresponding labels as training set, 10, 000 as validation set, and 10, 000 as testing set.…”

Section: Diversity Regularized Natural Language Inferencementioning

confidence: 99%

“…50. The divergence was measured using the Wasserstein distance measure [160] and it can be observed that the regularizing effect of divReg stabilizes the adversarial training and prevents mode collapse. For unregularized network, however, the mode started to collapse around 45th epoch.…”

Section: Diversity Regularized Adversarial Learning (Direal)mentioning

confidence: 99%

See 2 more Smart Citations

Receptive fields optimization in deep learning for enhanced interpretability, diversity, and resource efficiency.

Ayinde¹

View full text Add to dashboard Cite

show abstract

Stacked dynamic target regularization enhanced autoencoder for soft sensor in industrial processes

Guo,

Zhao,

2024

Can J Chem Eng

View full text Add to dashboard Cite

Stacked autoencoders (SAEs) have great potential in developing soft sensors due to their excellent feature extraction capabilities. However, the pre‐training stage of SAE is unsupervised and some important information related to target variables may be discarded. Meanwhile, as the depth of the network increases, reconstruction errors continue to accumulate, resulting in incomplete feature representations of the original input. In addition, the dynamic nature of the data affects the predictive results of the model. To address these issues, the stacked dynamic target regularization enhanced autoencoder (SDTR‐EAE) method is proposed, which adds the DTR and the original input information layer by layer to enhance the feature extraction. To adapt to the dynamic changes in data and extract target‐related features, entropy weight grey relational analysis (EW‐GRA) is used as the DTR term to constrain the weight matrix and suppress irrelevant features. To reduce the accumulation of information loss during the reconstruction, an information enhancement layer is introduced, where the original inputs and the information of the hidden layers of previous DTR‐EAE units are added to the follow‐up DTR‐EAE unit. Finally, in the regression process, the DTR term is used again to fully utilize depth features for quality prediction and prevent overfitting. Experimental verifications using the debutanizer column and thermal power plant are conducted to validate the effectiveness of the proposed modelling method.

show abstract

Regularizing Deep Neural Networks by Enhancing Diversity in Feature Extraction

Cited by 65 publications

References 20 publications

An extreme rainfall-induced landslide susceptibility assessment using autoencoder combined with random forest in Shimane Prefecture, Japan

An extreme rainfall-induced landslide susceptibility assessment using autoencoder combined with random forest in Shimane Prefecture, Japan

Receptive fields optimization in deep learning for enhanced interpretability, diversity, and resource efficiency.

Stacked dynamic target regularization enhanced autoencoder for soft sensor in industrial processes

Contact Info

Product

Resources

About