Regularization and Semi-supervised Learning on Large Graphs

Abstract

Abstract. We consider the problem of labeling a partially labeled graph. This setting may arise in a number of situations from survey sampling to information retrieval to pattern recognition in manifold settings. It is also of potential practical importance, when the data is abundant, but labeling is expensive or requires human assistance. Our approach develops a framework for regularization on such graphs. The algorithms are very simple and involve solving a single, usually sparse, system of linear equations. Using the notion of algorithmic stability, we derive bounds on the generalization error and relate it to structural invariants of the graph. Some experimental results testing the performance of the regularization algorithm and the usefulness of the generalization bound are presented.

Stable Transductive Learning

“…When the uniform stability obtains lower values the algorithm performs very poorly. This may indicate that a good uniform stability is correlated with degenerated behavior (similar phenomenon was observed in [2]). In contrast, we see that very good weak stability can coincide with very high performance.…”

“…Stability was first considered in the context of transductive learning by Belkin et al [2]. There the authors applied uniform inductive stability notions and results of [4] to a specific graph-based transductive learning algorithm.…”

“…2 In applications of the 0/1 loss function we always apply the sign function on the soft classification A S m ,X u (x). When using the margin loss function we denote the training and transductive errors of A byR γ m (A) and R γ u (A), respectively.…”

On the relation between multi-instance learning and semi-supervised learning

“…Many semi-supervised learning approaches have been developed. Some approaches use a generative model for the classifier and employ EM to model the label estimation or parameter estimation process (Miller & Uyar, 1997;Nigam et al, 2000); some approaches use the unlabeled data to regularize the learning process in various ways, e.g., defining a graph on the data set and then enforcing the label smoothness over the graph as a regularization term (Belkin et al, 2001;Zhou et al, 2005a;Zhu et al, 2003); some approaches train two learners and then let the learners to label unlabeled instances for each other (Blum & Mitchell, 1998;Goldman & Zhou, 2000;Zhou & Li, 2005).…”

A neural network algorithm for semi-supervised node label learning from unbalanced data

“…Other approaches are based on graph regularization (Belkin et al, 2004;Delalleau et al, 2005), and on the exploitation of the properties of the graph Laplacian associated to the weight matrix of the graph (Belkin & Niyogi, 2003). Methods based on the amount of functional flow through the nodes (Nabieva et al, 2005), on global graph consistency (Vazquez et al, 2003;Karaoz et al, 2004), on Markov and Gaussian Random Fields (Tsuda et al, 2005;Mostafavi et al, 2008), and recently on kernelized score functions have been applied to the predic-tion of gene functions.…”