Taskonomy: Disentangling Task Transfer Learning

Abstract

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity.We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2 3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.

Multitask Learning to Improve Egocentric Action Recognition

“…MTL has been applied to computer vision problems such as joint object and action detection [19], object detection and segmentation [14] and boundary, surface normal and saliency estimation together with object segmentation and detection [23]. In [57], the relationship between tasks is modelled in a latent space to transfer knowledge between them and reduce the number of required training samples. MTL in egocentric vision appears in [1,28,25,18,29,47].…”

Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum

“…Up until now, most detection and segmentation tasks have relied on ImageNet [63] fine-tuning [13,87]. With fine-tuning, learned parameters or features of source tasks may be forgotten after learning target tasks [29], and domain similarity between tasks being important for transfer learning [89]. Furthermore, transferring knowledge between dissimilar tasks may cause negative transfer [62,79].…”

Encoder–decoder network for pixel‐level road crack detection in black‐box images

“…Transfer learning is a method for imparting the knowledge of models trained with large amounts of data to the models in other domains by using pretrained weights with large‐size data sets, instead of using randomly initialized weights of CNN. This learning method not only accelerates the convergence of CNN but also improves the performance of the model trained with small amount of training data (Zamir et al., ). It is successful at solving computer vision problems in domains such as construction equipment detection (Kim, Kim, Hong, & Byun, ), pavement crack detection (Gopalakrishnan et al., ; Zhang et al., ; Zhang et al., ), baggage recognition using X‐ray images (Akçay, Kundegorski, Devereux, & Breckon, ), and saliency prediction in natural videos (Chaabouni, Benois‐Pineau, & Amar, ) with relatively little training data.…”