A Principled Approach for Learning Task Similarity in Multitask Learning

Abstract

Multitask learning aims at solving a set of related tasks simultaneously, by exploiting the shared knowledge for improving the performance on individual tasks. Hence, an important aspect of multitask learning is to understand the similarities within a set of tasks. Previous works have incorporated this similarity information explicitly (e.g., weighted loss for each task) or implicitly (e.g., adversarial loss for feature adaptation), to achieve good empirical performances. However, the theoretical motivations for adding task similarity knowledge are often missing or incomplete. In this paper, we give a different perspective from a theoretical point of view to understand this practice. We first provide an upper bound on the generalization error of multitask learning, showing the benefit of explicit and implicit task similarity knowledge. We systematically derive the bounds based on two distinct task similarity metrics: H divergence and Wasserstein distance. From these theoretical results, we revisit the Adversarial Multitask Neural Network, proposing a new training algorithm to learn the task relation coefficients and neural network parameters iteratively. We assess our new algorithm empirically on several benchmarks, showing not only that we find interesting and robust task relations, but that the proposed approach outperforms the baselines, reaffirming the benefits of theoretical insight in algorithm design.

Multitask Online Mirror Descent

“…In [29] they look at distributed online classification in a synchronous setting. Other approaches for learning task similarities have been investigated in [42,31,36].…”

Multi-task Learning for Intrusion Detection and Analysis of Computer Network Traffic

“…This binding is quantified using a real number, and therefore the problem setting was regression. As for task similarity, although there is existing work that focuses on learning that similarity [10,11], the reported method exploits the amino-acid sequence of each protein and uses it to compute the similarity between proteins. This similarity is then assumed to be the task similarity and the approach combines datasets from various proteins into one large dataset and adds the similarity values as new features.…”

Probabilistic task modelling for meta-learning

“…Another approach is to formulate multitask learning as a convex optimisation problem either to cluster tasks and utilise the clustering results to fast track the learning (Jacob et al, 2009), or to learn task relationship through task covariance matrices (Zhang and Yeung, 2012). Other approaches provided theoretical guarantees when learning the similarity or relationship between tasks (Shui et al, 2019). Recently, the taskonomy project (Zamir et al, 2018) was conducted to carry out extensive experiments on 26 computer-vision tasks to empirically analyse the correlation between those tasks.…”