Label Propagation Adaptive Resonance Theory for Semi-Supervised Continuous Learning

Abstract

Semi-supervised learning and continuous learning are fundamental paradigms for human-level intelligence. To deal with real-world problems where labels are rarely given and the opportunity to access the same data is limited, it is necessary to apply these two paradigms in a joined fashion. In this paper, we propose Label Propagation Adaptive Resonance Theory (LPART) for semi-supervised continuous learning. LPART uses an online label propagation mechanism to perform classification and gradually improves its accuracy as the observed data accumulates. We evaluated the proposed model on visual (MNIST, SVHN, CIFAR-10) and audio (NSynth) datasets by adjusting the ratio of the labeled and unlabeled data. The accuracies are much higher when both labeled and unlabeled data are used, demonstrating the significant advantage of LPART in environments where the data labels are scarce.

Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning

“…The performance of the competitive models such as A-SOINN (Shen et al 2011) and LPART (Kim et al 2020) was compared to MPART using various message passing layers L and 'Explorer' strategy in Table 2. The multi-layer perceptron (MLP) model was used as a reference of fully supervised learning, which was trained using all labeled data for each dataset.…”

“…However, most of these methods are not suitable for online learning, because they need predefined topological information or whole training data repeatedly. LPART (Kim et al 2020) uses online label propagation on the ART network trained in semi-supervised manner to overcome this issue, but the conveyed label information is limited between the nodes due to the weak topology.…”

“…Uncertainty Estimation We use two different metrics to measure the uncertainty of the nodes as in (Kim et al 2020). The first uncertainty, u J,a , can be seen as an aleatoric uncertainty which is measured by the entropy of the classification probability as shown in Equation 5.…”

“…In order for MPART to properly form clusters and generate a topological graph for high-dimensional input data, we used a pre-trained feature extractor for dimension reduction. Any typical feature extractor can be applied, and here we employed an encoder of a variant of VAE (Kingma and Welling 2013), which is adapted to weakly-supervised training as in (Kim et al 2020). This feature extractor is trained with weak labels indicating whether or not the class of the current sample is the same as that of the previous one.…”

RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Data

“…For example, we assume the prior class distribution p(y) is uniform over all classes. One can extend RoPAWS to class-imbalanced setup (Kim et al, 2020;Wei et al, 2021) by assuming non-uniform prior. RoPAWS can also handle label shift, i.e., class distribution of labeled and unlabeled data are different.…”