Untitled

Abstract

Multi-task learning (MTL) has achieved success over a wide range of problems, where the goal is to improve the performance of a primary task using a set of relevant auxiliary tasks. However, when the usefulness of the auxiliary tasks w.r.t. the primary task is not known a priori, the success of MTL models depends on the correct choice of these auxiliary tasks and also a balanced mixing ratio of these tasks during alternate training. These two problems could be resolved via manual intuition or hyper-parameter tuning over all combinatorial task choices, but this introduces inductive bias or is not scalable when the number of candidate auxiliary tasks is very large. To address these issues, we present AUTOSEM, a two-stage MTL pipeline, where the first stage automatically selects the most useful auxiliary tasks via a Beta-Bernoulli multi-armed bandit with Thompson Sampling, and the second stage learns the training mixing ratio of these selected auxiliary tasks via a Gaussian Process based Bayesian optimization framework. We conduct several MTL experiments on the GLUE language understanding tasks, and show that our AUTOSEM framework can successfully find relevant auxiliary tasks and automatically learn their mixing ratio, achieving significant performance boosts on several primary tasks. Finally, we present ablations for each stage of AUTOSEM and analyze the learned auxiliary task choices.

A Cascade Approach to Neural Abstractive Summarization with Content Selection and Fusion

“…The per-target-word loss is then interpolated with instance prediction (one or two sentences) loss using a coefficient λ. Such a multi-task learning objective has been shown to improve performance on a number of tasks (Guo et al, 2019).…”

GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks

“…with Bayesian optimization (Ruder and Plank, 2017), but these methods are time-and resource-consuming due to their reliance on multitask experiments involving all the candidate tasks. AUTOSEM (Guo et al, 2019) combines the two settings into one method, selecting candidate tasks with Thompson sampling and deciding the ratio with which to draw training instances from the selected tasks via a Gaussian Process. Despite the higher quality of the auxiliary task sets it generates, AUTOSEM is still costly, similar to Ruder and Plank (2017).…”

“…The contributions of this paper are three-fold: Following Guo et al (2019), we use the 8 classification tasks in GLUE benchmarks (Wang et al, 2019), namely CoLA, MRPC, MNLI, QNLI, QQP, RTE, SST-2, and WNLI, in our main experiments. We apply the standard split of these datasets as Wang et al (2019) describe.…”