Identifying beneficial task relations for multi-task learning in deep neural networks

Bingel, Joachim; Søgaard, Anders

doi:10.48550/arxiv.1702.08303

Cited by 25 publications

(28 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two early concurrent works of learning to group tasks are (Alonso and Plank, 2016) and (Bingel and Søgaard, 2017).…”

Section: Grouping Tasksmentioning

confidence: 99%

Multi-Task Learning with Deep Neural Networks: A Survey

Crawshaw

2020

Preprint

118

View full text Add to dashboard Cite

Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. Such approaches offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information. However, the simultaneous learning of multiple tasks presents new design and optimization challenges, and choosing which tasks should be learned jointly is in itself a non-trivial problem. In this survey, we give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field. Our discussion is structured according to a partition of the existing deep MTL techniques into three groups: architectures, optimization methods, and task relationship learning. We also provide a summary of common multi-task benchmarks.

show abstract

“…Two early concurrent works of learning to group tasks are (Alonso and Plank, 2016) and (Bingel and Søgaard, 2017).…”

Section: Grouping Tasksmentioning

confidence: 99%

Multi-Task Learning with Deep Neural Networks: A Survey

Crawshaw

2020

Preprint

118

View full text Add to dashboard Cite

show abstract

“…Then, either the parameters of the models are encouraged to be similar by regularizing a distance [29,30] or the knowledge of tasks are linearly combined to produce output for each task [31]. Other works try to improve the multi-task performance by addressing what to share [32,33,34], which tasks to train together [35,36], or inferring task-specifc model weights [37,30,38]. Recently, the transference is proposed to analyze the information transfer in a general MTL framework [2].…”

Section: Related Workmentioning

confidence: 99%

Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning

Abdollahzadeh¹,

Malekzadeh²,

Cheung³

2021

Preprint

View full text Add to dashboard Cite

Multimodal meta-learning is a recent problem that extends conventional few-shot meta-learning by generalizing its setup to diverse multimodal task distributions. This setup makes a step towards mimicking how humans make use of a diverse set of prior skills to learn new skills. Previous work has achieved encouraging performance. In particular, in spite of the diversity of the multimodal tasks, previous work claims that a single meta-learner trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions. The improvement is attributed to knowledge transfer between different modes of task distributions. However, there is no deep investigation to verify and understand the knowledge transfer between multimodal tasks. Our work makes two contributions to multimodal meta-learning. First, we propose a method to quantify knowledge transfer between tasks of different modes at a micro-level. Our quantitative, task-level analysis is inspired by the recent transference idea from multi-task learning. Second, inspired by hard parameter sharing in multi-task learning and a new interpretation of related work, we propose a new multimodal meta-learner that outperforms existing work by considerable margins. While the major focus is on multimodal meta-learning, our work also attempts to shed light on task interaction in conventional meta-learning. The code for this project is available at https://miladabd.github.io/KML.Multimodal Model-Agnostic Meta-Learning (MMAML) [1] proposes a framework to better handle multimodal task distributions and achieves encouraging performance. As one of the most intriguing findings, MMAML claims that a single meta-learner trained on a multimodal distribution can sometimes outperform multiple specialized meta-learners trained on individual unimodal distributions. This was observed in spite of the diversity of the multimodal tasks. In [1], this observation is attributed to knowledge transfer across different modes of multimodal task distribution.

show abstract

“…On the problem of identifying task relatedness, Ben-David and Schuller (2003) provided a formal framework for task relatedness and derived generalization error bounds for learning of multiple tasks. Bingel and Søgaard (2017) explored task relatedness via exhaustively experimenting with all possible two task tuples in a nonautomated multi-task setup. Other related works explored data selection, where the goal is to select or reorder the examples from one or more domains (usually in a single task) to either improve the training efficiency or enable better transfer learning.…”

Section: Related Workmentioning

confidence: 99%

AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning

Han¹,

Pasunuru²,

Bansal³

2019

Preprint

View full text Add to dashboard Cite

Multi-task learning (MTL) has achieved success over a wide range of problems, where the goal is to improve the performance of a primary task using a set of relevant auxiliary tasks. However, when the usefulness of the auxiliary tasks w.r.t. the primary task is not known a priori, the success of MTL models depends on the correct choice of these auxiliary tasks and also a balanced mixing ratio of these tasks during alternate training. These two problems could be resolved via manual intuition or hyper-parameter tuning over all combinatorial task choices, but this introduces inductive bias or is not scalable when the number of candidate auxiliary tasks is very large. To address these issues, we present AUTOSEM, a two-stage MTL pipeline, where the first stage automatically selects the most useful auxiliary tasks via a Beta-Bernoulli multi-armed bandit with Thompson Sampling, and the second stage learns the training mixing ratio of these selected auxiliary tasks via a Gaussian Process based Bayesian optimization framework. We conduct several MTL experiments on the GLUE language understanding tasks, and show that our AUTOSEM framework can successfully find relevant auxiliary tasks and automatically learn their mixing ratio, achieving significant performance boosts on several primary tasks. Finally, we present ablations for each stage of AUTOSEM and analyze the learned auxiliary task choices.

show abstract

Identifying beneficial task relations for multi-task learning in deep neural networks

Cited by 25 publications

References 0 publications

Multi-Task Learning with Deep Neural Networks: A Survey

Multi-Task Learning with Deep Neural Networks: A Survey

Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning

AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning

Contact Info

Product

Resources

About