Multi-Task Learning for Dense Prediction Tasks: A Survey

Abstract

With the advent of deep learning, many dense prediction tasks, i.e. tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.

A Deep Transfer Learning Method for Bearing Fault Diagnosis Based on Domain Separation and Adversarial Learning

“…During the training process, the decreasing rate of L c , L d , L s , L p , and L r may be inconsistent, causing the model to be dominated by a certain module [14]. Hence, need to use weight coefficients, for the entire network, the final loss function is given by…”

Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection

“…It achieves this by utilizing encoder-level interactions to generate a shared representation [22,9,17], by using decoder-level interactions to improve single task results from multi-modal distillation [35,38], or a set combination of both. [32] shows that in an MTL setting, performance strongly varies depending on a wide range of parameters (e.g task type, label source) and thus architecture and optimization strategies must be selected on a per case basis. In general it is observed that encoder level interactions perform well for multiple classification problems while decoder level interactions have an advantage in dense prediction tasks.…”

“…The surrounding area of each point is discretized into a set number of bins. This allows us to restate the problem of the bounding box center localization on the transverse plane (x, z) as a classification problem which are shown to be better fitted for encoder-focused architectures [32]. To achieve finer details, we allow a residual to be regressed for each bin.…”

“…This can be highly beneficial when dealing with real world problems of navigation in complex scenes that involves onthe-go path planning and collision avoidance. Semantic Feature Fusion: In MTL, the problem of invariance vs. sensitivity is always apparent [32]. The network may never converge to a state where it extracts a vital feature for one task because it contradicts with the objective of the other.…”

JMRPE‐Net: Joint modulation recognition and parameter estimation of cognitive radar signals with a deep multitask network

“…Multitask learning (MTL) deep neural networks are widely used in many fields [26,[33][34][35][36]. It would be an intuitive and promising idea to concurrently solve the AMR and MPE tasks together in an MTL way.…”