NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction

Abstract

In this paper, we propose a novel Convolutional Neural Network (CNN) structure for general-purpose multi-task learning (MTL), which enables automatic feature fusing at every layer from different tasks. This is in contrast with the most widely used MTL CNN structures which empirically or heuristically share features on some specific layers (e.g., share all the features except the last convolutional layer). The proposed layerwise feature fusing scheme is formulated by combining existing CNN components in a novel way, with clear mathematical interpretability as discriminative dimensionality reduction, which is referred to as Neural Discriminative Dimensionality Reduction (NDDR). Specifically, we first concatenate features with the same spatial resolution from different tasks according to their channel dimension. Then, we show that the discriminative dimensionality reduction can be fulfilled by 1 × 1 Convolution, Batch Normalization, and Weight Decay in one CNN. The use of existing CNN components ensures the end-to-end training and the extensibility of the proposed NDDR layer to various state-of-the-art CNN architectures in a "plug-andplay" manner. The detailed ablation analysis shows that the proposed NDDR layer is easy to train and also robust to different hyperparameters. Experiments on different task sets with various base network architectures demonstrate the promising performance and desirable generalizability of our proposed method. The code of our paper is available at https

Learning Shared Knowledge for Deep Lifelong Learning using Deconvolutional Networks

“…. , Z Tmax , and must learn a model (i.e., a classifier) for each task [Chen and Liu, 2016]. The system has no a priori information about the task distribution, order, or total number of tasks T max .…”

“…Similar to the work of dynamic filter generation [Jia et al, 2016;Ha et al, 2017], a single operation can be employed to expand the knowledge base into a large task-specific filter. However, in our proposal, deconvolution and tensor contraction are used instead as a two-staged expansion, distinguishing between the transfer process along the spatial axis of the images and along the channels of the images.…”

“…Much of the success of deep discriminative learning, including visual recognition [Krizhevsky et al, 2012;Simonyan and Zisserman, 2015;He et al, 2016] and image segmentation, stems from the remarkable ability of neural networks to generalize to unseen data that is drawn from the same underlying data generating process as the training set. Comparatively less success has been achieved when deep networks are deployed in an online multi-task learning (MTL) or lifelong learning fashion [Chen and Liu, 2016].…”

“…Although effective in providing flexible transfer, this approach is currently limited to learning with shallow models. Deep MTL methods such as tensor factorization [Yang and Hospedales, 2017], progressive neural networks , and dynamic filters [Jia et al, 2016;Ha et al, 2017] relate multiple deep networks and permit transfer, but do not provide sufficiently flexible transfer mechanisms nor support all characteristics (such as reverse transfer) needed for lifelong learning.…”

SIGNet: Semantic Instance Aided Unsupervised 3D Geometry Perception

“…However, this work is fully supervised and only uses sub-optimal traditional methods. Wang et al [44], Cross-Stitching [35], UberNet [23] and NDDR-CNN [12] all report improved performance over single-task baselines. But they have not addressed outdoor scenes and unsupervised geometry understanding.…”