AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning

Han, Guohong; Pasunuru, Ramakanth; Bansal, Mohit

doi:10.48550/arxiv.1904.04153

Cited by 5 publications

(7 citation statements)

References 32 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because neural networks are prone to overfitting, that is, they tend to even memorize the random labels in the training set or mostly focus only on obvious data features; auxiliary tasks help the models learn the features that are overlooked, but meaningful. The choice of auxiliary tasks to use and avert negative transfers has been an active area of research in multi-task learning [5,19,20].…”

Section: Self-supervision Tasks and Auxiliary Task Learningmentioning

confidence: 99%

Trainable Weights for Multitask Learning

Ryu,

Lee,

Choi

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The research on multi-task learning has been steadily increasing due to its advantages, such as preventing overfitting, averting catastrophic forgetting, solving multiple inseparable tasks, and coping with data shortage. Here, we question whether to incorporate different orderings of feature levels based on distinct characteristics of tasks and their interrelationships in multitask learning. While in many classification tasks, leveraging the features extracted from the last layer is common, we hypothesized that given different characteristics of tasks there might be a need to encompass different representation levels, i.e., different orderings of feature levels. Hence, we utilized the knowledge of different representation levels by features extracted from the various blocks of the main module and applied trainable parameters as weights on the features. This indicates that we optimized the solution to the question by learning to weigh the features in a task-specific manner and solving tasks with a combination of newly weighted features. Our method SimPara presents a modular topology of multitask learning that is efficient in terms of memory and computation, effective, and easily applicable to diverse tasks or models. To show that our approach is taskagnostic and highly applicable, we demonstrate its effectiveness in auxiliary task learning, active learning, and multilabel learning settings. This work underscores that by simply learning weights to better order the features learned by a single backbone, we can incur better task-specific performance of the model.

show abstract

Section: Self-supervision Tasks and Auxiliary Task Learningmentioning

confidence: 99%

Trainable Weights for Multitask Learning

Ryu,

Lee,

Choi

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…(3) Efficient intermediate task selection for pre-finetuning. Based on the transferability among different tasks, researchers also explore how to efficiently choose the most appropriate combinations of intermediate tasks from an abundance of candidate tasks through embedding-based methods [535], manually-defined features [532], task gradients [536] and Beta-Bernoulli multi-armed bandit [537]. (4) The power of scale for multi-task pre-finetuning.…”

Section: Multi-task Learningmentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

View full text Add to dashboard Cite

domains indexed by Google News. It contains 31 million documents with an average length of 793 BPE tokens. Like C4, it excludes examples with duplicate URLs. News dumps from December 2016 through March 2019 were used as training data, articles published in April 2019 from the April 2019 dump were used for evaluation. OpenWebText2(OWT2). OWT2 is an enhanced version of the original OpenWebTextCorpus, including content from multiple languages, document metadata, multiple dataset versions, and open source replication code, covering all Reddit submissions from 2005 up until April 2020. PubMed Central(PMC). PMC is a free full-text archive of biomedical and life sciences journal literature from the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The dataset is updated daily. In addition to full-text articles, they contain corrections, retractions, and expressions of concern, as well as file lists that include metadata for articles in each dataset.PMC obtained by open registration in Amazon Web Services (AWS) includes The PMC Open Access Subset and The Author Manuscript Dataset. The PMC Open Access Subset includes all articles and preprints in PMC with a machine-readable Creative Commons license that allows reuse. The Author Manuscript Dataset includes accepted author manuscripts collected under a funder policy in PMC and made available in machine-readable formats for text mining. ArXiv. ArXiv is a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more. It provides open access to academic articles, covering many subdisciplines from vast branches of physics to computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics, which is helpful to the potential downstream applications of the research field. In addition, the writing language of LaTeX also contributes to the study of language models. Colossal Clean Crawled Corpus(C4). C4 is a colossal, cleaned version of Common Crawl's web crawl corpus. It is based on Common Crawl dataset and was used to train the T5 text-to-text Transformer models. The cleaned English version of C4 has 364,868,901 training examples and 364,608 validation examples, while the uncleaned English version has 1,063,805,324 training examples and 1,065,029 validation examples; the realnewslike version has 13,799,838 training examples and 13,863 validation examples, while the webtextlike version has 4,500,788 training examples and 4,493 validation examples. Wiki-40B. Wikipedia (Wiki-40B) is a clean-up text collection containing more than 40 Wikipedia language editions of pages corresponding to entities. The dataset is split into train/validation/test sets for each language. The training set has 2,926,536 examples, the validation set has 163,597 examples, and the test set has 162,274 examples. Wiki-40B is cleaned by a page filter to remove ambiguous, redirected, deleted, and non-physical pages. CLUECorpus2020. CLUECorpus2020 ...

show abstract

“…Optimization techniques. Guo et al (2019) use ideas from the multi-armed bandit literature to develop a method for weighting each task. Compared to their method, our SVD-based method is conceptually simpler and requires much less computation.…”

Section: Related Workmentioning

confidence: 99%

Understanding and Improving Information Transfer in Multi-Task Learning

Wu,

Zhang,

Ré

2020

Preprint

View full text Add to dashboard Cite

We investigate multi-task learning approaches that use a shared feature representation for all tasks. To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT LARGE using our alignment method. We also design an SVD-based task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.

show abstract

AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning

Cited by 5 publications

References 32 publications

Trainable Weights for Multitask Learning

Trainable Weights for Multitask Learning

A Roadmap for Big Model

Understanding and Improving Information Transfer in Multi-Task Learning

Contact Info

Product

Resources

About