GAR: An efficient and scalable graph-based activity regularization for semi-supervised learning

Abstract

In this paper, we propose a novel graph-based approach for semi-supervised learning problems, which considers an adaptive adjacency of the examples throughout the unsupervised portion of the training. Adjacency of the examples is inferred using the predictions of a neural network model which is first initialized by a supervised pretraining. These predictions are then updated according to a novel unsupervised objective which regularizes another adjacency, now linking the output nodes. Regularizing the adjacency of the output nodes, inferred from the predictions of the network, creates an easier optimization problem and ultimately provides that the predictions of the network turn into the optimal embedding. Ultimately, the proposed framework provides an effective and scalable graph-based solution which is natural to the operational mechanism of deep neural networks. Our results show comparable performance with state-of-the-art generative approaches for semi-supervised learning on an easier-to-train, low-cost framework.

Deep learning-based clustering approaches for bioinformatics

“…More recent approaches include convolutional embedded networks (CENs) [ 30 ], deep convolutional embedded clustering (DCEN) [ 29 ], discriminatively boosted clustering (DBC) [ 47 ], CNN-based joint clustering and RL with feature drift compensation (UMMC) [ 51 ], deep continuous clustering (DCC) [ 49 ], learning latent representations for clustering (IMSAT) [ 50 ] and deep adaptive clustering (DAC) [ 48 ]. We listed reviewed approaches including links to the original papers and their implementations that can be found at https://github.com/rezacsedu/Deep-learning-for-clustering-in-bioinformatics .…”

The global biogeography of amino acid variants within a single SAR11 population is governed by natural selection

“…sub-classes, under each one of parents [45]. ACOL is a novel output layer modification for deep neural networks to allow simultaneous supervised classification (per provided parent-classes) and unsupervised clustering (within each parent) where clustering is performed with a Graph-based Activity Regularization (GAR) technique recently proposed in [44]. More specifically, as ACOL duplicates the softmax nodes at the output layer for each class, GAR allows for competitive learning between these duplicates on a traditional error-correction learning framework.…”

“…To estimate an unbiased distance between our metagenomes based on SAAVs, we used a novel deep neural network modification called the auto-clustering output layer (ACOL). Briefly, ACOL relies on a recently introduced graph-based activity regularization (GAR) technique for competitive learning from hyper-dimensional data to demarcate fine clusters within user-defined 'parent' classes [44]. In this application of ACOL, however, we modified the algorithm so it can reveal latent groups in our SAAVs in a fully unsupervised manner through frequent random sampling of SAAVs to create pseudo-parent class labels instead of user-defined classes [45].…”

Least squares support vector machine with self-organizing multiple kernel learning and sparsity

“…x x (10) where Cauchy σ is an adjustable parameter. The Cauchy kernel is a long-tailed kernel, deriving from the Cauchy distribution.…”

“…For different types of modeling methods, machine learning can be classified as unsupervised learning, semi-supervised learning, and supervised learning [9][10][11]. The focus of this paper is on supervised learning methods.…”