When is multitask learning effective? Semantic sequence prediction under varying data conditions

Abstract

Multitask learning has been applied successfully to a range of tasks, mostly morphosyntactic. However, little is known on when MTL works and whether there are data characteristics that help to determine its success. In this paper we evaluate a range of semantic sequence labeling tasks in a MTL setup. We examine different auxiliary tasks, amongst which a novel setup, and correlate their impact to datadependent conditions. Our results show that MTL is not always effective, significant improvements are obtained only for 1 out of 5 tasks. When successful, auxiliary tasks with compact and more uniform label distributions are preferable.

Improving sentiment analysis with multi-task learning of negation

“…Both the SFU and CD data sets not only have low label kurtosis but also have relatively low label entropy. This partially aligns with previous research (Martínez Alonso and Plank 2017; Bingel and Søgaard 2017), which suggests that for an auxiliary task to improve the main task, the entropy of the labels should be high (implying the task should not be trivial to learn), and the kurtosis should be low (the labels should not have an overly long-tailed distribution). The fact that the multi-word task is more helpful, however, seems to indicate that the appropriateness of the auxiliary task for the main task is more important than the specific data set properties.…”

“…Here, entropy indicates the amount of uncertainty in the label distribution, while kurtosis indicates the skewness. These measures have been shown to correlate well with the usefulness of auxiliary tasks in previous work (Martínez Alonso and Plank 2017; Bingel and Søgaard 2017).…”

“…When the main task and auxiliary task are closely related, this approach has been shown to be an effective way to improve model performance (Collobert et al . 2011; Peng and Dredze 2017; Martínez Alonso and Plank 2017; Augenstein, Ruder, and Søgaard 2018). Other research (Søgaard and Goldberg 2016), on the other hand, finds that it is better to make predictions for low-level auxiliary tasks at lower layers of a multi-layer MTL setup.…”

“…Some research suggests high-level semantic auxiliary tasks generally help more than low-level auxiliary tasks as MTL tends to work when the main task learning plateaus quickly and the auxiliary task learning does not (Bingel and Søgaard 2017). Others find that auxiliary tasks with compact, uniform label distributions are preferable (Martínez Alonso and Plank 2017). Additionally, choosing a suitable auxiliary task is still vital, as introducing an unsuitable auxiliary task can actually hurt performance (Augenstein and Søgaard 2017).…”

Mining arguments in scientific abstracts with discourse-level embeddings

“…When jointly training argument mining and discourse parsing tasks, in contrast, the results obtained for the argument mining models are worse than those obtained when the models are trained in single task settings. This effect, known as negative transfer, is not uncommon in multi-task settings [29]. In fact, multi-task learning architectures are known to be sensitive to a large number of parameters, including the distribution of the labels, the sizes of the respective datasets and the sampling strategies implemented in order to select the mini-batches when switching between tasks [54].…”