Deep Neural Network Self-training Based on Unsupervised Learning and Dropout

Abstract

In supervised learning methods, a large amount of labeled data is necessary to find reliable classification boundaries to train a classifier. However, it is hard to obtain a large amount of labeled data in practice and it is time-consuming with a lot of cost to obtain labels of data. Although unlabeled data is comparatively plentiful than labeled data, most of supervised learning methods are not designed to exploit unlabeled data. Self-training is one of the semisupervised learning methods that alternatively repeat training a base classifier and labeling unlabeled data in training set. Most self-training methods have adopted confidence measures to select confidently labeled examples because high-confidence usually implies low error. A major difficulty of self-training is the error amplification. If a classifier misclassifies some examples and the misclassified examples are included in the labeled training set, the next classifier may learn improper classification boundaries and generate more misclassified examples. Since base classifiers are built with small labeled dataset and are hard to earn good generalization performance due to the small labeled dataset. Although improving training procedure and the performance of classifiers, error occurrence is inevitable, so corrections of self-labeled data are necessary to avoid error amplification in the following classifiers. In this paper, we propose a deep neural network based approach for alleviating the problems of self-training by combining schemes: pre-training, dropout and error forgetting. By applying combinations of these schemes to various dataset, a trained classifier using our approach shows improved performance than trained classifier using common self-training.

Semi-supervised Segmentation of Salt Bodies in Seismic Images Using an Ensemble of Convolutional Neural Networks

“…Every next round, we first (a) retrain the model using jointly ground truth labels and confident pseudo-labels; and then (b) update the pseudo-labels for all unlabeled data using the new model. It is crucial to reset model weights before every round of self-training not to accumulate errors in pseudo-labels during multiple rounds [29].…”

“…Average voting scheme corrects examples which could be mislabeled by one of the models, hence facilitates more reliable pseudo-labeling. Moreover, to further reduce the error amplification we retrain our models from scratch and predict labels for all unlabeled examples every round in similar spirit as [29].…”

A Deep Learning Approach to Network Intrusion Detection

“…Lee et al [28] propose a deep-learning approach to fault monitoring in semiconductor manufacturing. They use a Stacked de-noising Auto-encoder (SdA) approach to provide an unsupervised learning solution.…”

Comparisons of Deep Learning Algorithms for MNIST in Real-Time Environment

“…(3). We added tiny change to the squashing function given in [2], it is shown in Eq. (4), and since v j output of a capsule j and s j might share 0 vector.…”

“…Especially, deep neural networks have achieved great success in various applications [1,2], particularly in tasks involving visual information. There have introduced many state-of-the-art models in the field that perform dissimilar tasks with a high accuracy and very effectively.…”