No abstract

No abstract

Unsupervised feature learning via auto-encoders results in low-dimensional representations in latent space that capture the patterns of input data. The auto-encoders with robust regularization learn qualified features that are less sensitive to small perturbations of inputs. However, the previous robust auto-encoders highly depend on pre-defined structure settings and often learn full-connected networks that are easily prone to over-fitting. To solve the above limitations, we propose in this paper an explicitly regularized framework which improves the sparsity and flexibility of robust auto-encoders. First, our model encourages the activation functions to automatically adjust themselves between linear and non-linear ones. Second, the mapping functions of the encoder are constrained by group sparsity and exclusive sparsity to reduce the redundancy of parameters. The proximal gradient method is used to optimize our model since the objective function contains non-smooth components. We conduct experiments in single-layer and multiplelayer auto-encoders in the classification task. The numerical results show that our model achieves better accuracy than baseline models. Our method also shows better performance in denoising task.

Graph convolutional networks have made great progress in graph-based semi-supervised learning. Existing methods mainly assume that nodes connected by graph edges are prone to have similar attributes and labels, so that the features smoothed by local graph structures can reveal the class similarities. However, there often exist mismatches between graph structures and labels in many real-world scenarios, where the structures may propagate misleading features or labels that eventually affect the model performance. In this paper, we propose a multi-task self-distillation framework that injects self-supervised learning and self-distillation into graph convolutional networks to separately address the mismatch problem from the structure side and the label side. First, we formulate a self-supervision pipeline based on pre-text tasks to capture different levels of similarities in graphs. The feature extraction process is encouraged to capture more complex proximity by jointly optimizing the pre-text task and the target task. Consequently, the local feature aggregations are improved from the structure side. Second, self-distillation uses soft labels of the model itself as additional supervision, which has similar effects as label smoothing. The knowledge from the classification pipeline and the self-supervision pipeline is collectively distilled to improve the generalization ability of the model from the label side. Experiment results show that the proposed method obtains remarkable performance gains under several classic graph convolutional architectures.

No abstract