UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory

Abstract

In this work we introduce a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture that is trained end-to-end. Such a universal network can act like a 'swiss knife' for vision tasks; we call this architecture an UberNet to indicate its overarching nature.We address two main technical challenges that emerge when broadening up the range of tasks handled by a single CNN: (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. Properly addressing these two problems allows us to train accurate predictors for a host of tasks, without compromising accuracy.Through these advances we train in an end-to-end manner a CNN that simultaneously addresses (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all of these tasks in 0.7 seconds per frame on a single GPU. A demonstration of this system can be found at cvn.

Visual pathways from the perspective of cost functions and multi-task deep neural networks

“…Finding an objective for such a broad and vague task appears futile so that it is easier to define a subset of tasks like figure ground segmentation, saliency and boundaries. A noteworthy implementation of such a decomposition is the recent DNN 'Uber-Net' (Kokkinos, 2016), which solves 7 vision related tasks (boundary, surface normals, saliency, semantic segmentation, semantic boundary and human parts detection) with a single multi-scale DNN network to reduce the memory footprint. It can be assumed that such a multi-task training improves convergence speed and better generalization to unseen data, something that already has been observed on other multi-task setups related to speech processing, vision and maze navigation (Bilen & Vedaldi, 2016;Caruana, 1998;Dietterich, Hild, & Bakiri, 1990, 1995Mirowski et al, 2016).…”

Face Normals "In-the-Wild" Using Fully Convolutional Networks

“…In particular, motivated by the success of deep learning to various tasks including object detection, dense semantic segmentation, and normal estimation of scenes [23,2,33,62] etc., we propose to exploit the available large scale facial databases captured both in controlled, as well as in unconstrained conditions [8,48] to train a fully convolutional deep network that maps image pixels to normals. More precisely, to acquire accurate ground truth of facial normals we synthesise images of faces created with the use of recently released Large-Scale 3D Facial Models (LSFM) [8] which contains facial shapes of individuals with diverse ethnicities and characteristics.…”

“…Discriminative estimation of normals has started to recently received increased attention [2,13,62,33,45,15]. One of the first methods was proposed in [70].…”

“…Most recently, another regression DCNN trained for surface estimation was proposed in [33]. This DCNN is a part of the so-called UberNet architecture which was proposed for jointly solving multiple image labelling tasks: such as detection of boundaries, saliency, semantic segmentation, human-parts, surface normals etc.…”

“…As in [33] we keep the task-specific memory and computation budget low by applying linear operations within these skip layers, and fuse skip-layer results through additive fusion with learnt weights.…”