Multi-Task Learning of Pairwise Sequence Classification Tasks over
            Disparate Label Spaces

Augenstein, Isabelle; Ruder, Sebastian; Søgaard, Anders

doi:10.18653/v1/n18-1172

Cited by 70 publications

(69 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance boost is particularly significant for the out-ofdomain setting, where sluice networks add more than 1 point in accuracy compared to hard parameter sharing and almost .5 compared to the strongest baseline on average, demonstrating that sluice networks are particularly useful to help a model generalize better. In contrast to previous studies on MTL (Martínez Alonso and Plank 2017;Bingel and Søgaard 2017;Augenstein, Ruder, and Søgaard 2018), our model also consistently outperforms single-task learning. Overall, this demonstrates that our meta-architecture for learning which parts of multi-task models to share, with a small set of additional parameters to learn, can achieve significant and consistent improvements over strong baseline methods.…”

Section: Training and Evaluationcontrasting

confidence: 93%

Latent Multi-Task Architecture Learning

Ruder

Bingel

Augenstein

et al. 2019

AAAI

Self Cite

250

165

View full text Add to dashboard Cite

Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amount of sharing, and (c) the appropriate relative weights of the different task losses. Recent work has addressed each of the above problems in isolation. In this work we present an approach that learns a latent multi-task architecture that jointly addresses (a)-(c). We present experiments on synthetic data and data from OntoNotes 5.0, including four different tasks and seven different domains. Our extension consistently outperforms previous approaches to learning latent architectures for multi-task problems and achieves up to 15% average error reductions over common approaches to MTL.

show abstract

Section: Training and Evaluationcontrasting

confidence: 93%

Latent Multi-Task Architecture Learning

Ruder

Bingel

Augenstein

et al. 2019

AAAI

Self Cite

250

165

View full text Add to dashboard Cite

show abstract

“…Unlike (Hashimoto et al 2017) and other previous work (Katiyar and Cardie 2017;Bekoulis et al 2018;Augenstein, Ruder, and Søgaard 2018), we do not learn label embeddings, meaning that the (supervised) output/prediction of a layer is not directly fed to the following layer through an embedding learned during training. Nonetheless, sharing embeddings and stacking hierarchical encoders allows us to share the supervision from each task along the full structure of our model and achieve state-of-the-art performance.…”

Section: Related Workmentioning

confidence: 99%

A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks

Sanh

Wolf

Ruder

2019

AAAI

Self Cite

180

170

View full text Add to dashboard Cite

Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.

show abstract

“…In addition, many works have attempted to learn the relationship through a matrix space or utilize additional regularization to increase the model learning capability. For example, Augenstein et al (2018) propose to leverage unlabeled or auxiliary data for better text classification. They first designed a label embedding layer to learn a relationship space between disparate labels.…”

Section: Multi-task Learningmentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

150

View full text Add to dashboard Cite

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both lowlevel feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.

show abstract

Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces

Cited by 70 publications

References 34 publications

Latent Multi-Task Architecture Learning

Latent Multi-Task Architecture Learning

A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Contact Info

Product

Resources

About