Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text

Lange, Lukas; Iurshina, Anastasiia; Adel, Heike; Strötgen, Jannik

doi:10.18653/v1/2020.repl4nlp-1.14

Cited by 14 publications

(19 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead, labeled data from a high-resource language is leveraged. A multilingual model can be trained on the target task in a high-resource language and afterwards, applied to the unseen target languages, such as for named entity recognition Hvingelby et al, 2020), reading comprehension , temporal expression extraction (Lange et al, 2020c), or POS tagging and dependency parsing (Müller et al, 2020). Hu et al (2020) showed, however, that there is still a large gap between low and high-resource setting.…”

Section: Multilingual Language Modelsmentioning

confidence: 99%

“…Gui et al (2017), Liu et al (2017), Kasai et al (2019), Grießhaber et al (2020) and learned domainindependent representations using adversarial training. Kim et al (2017), and Lange et al (2020c) worked with language-independent representations for cross-lingual transfer. These examples show the beneficial exchange of ideas between NLP and the machine learning community.…”

Section: Ideas From Low-resource Machine Learning In Non-nlp Communitiesmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

Hedderich¹,

Lange²,

Adel³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

116

View full text Add to dashboard Cite

Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional labeled data like data augmentation and distant supervision as well as transfer learning settings that reduce the need for target supervision. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low-resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research.

show abstract

Section: Multilingual Language Modelsmentioning

confidence: 99%

Section: Ideas From Low-resource Machine Learning In Non-nlp Communitiesmentioning

confidence: 99%

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

Hedderich¹,

Lange²,

Adel³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

116

View full text Add to dashboard Cite

show abstract

“…However, a fundamental limitation of existing crosslingual models for REE is the monolingual bias due to the sole reliance on source language data for training. In other NLP tasks, LADV has been explored to address this issue by leveraging unlabeled data in the target language to perform crosslingual representation alignment Huang et al, 2019;Lange et al, 2020;Cao et al, 2020;He et al, 2020). Unfortunately, LADV suffers from the cross-class alignment issue, making it less optimal for crosslingual REE.…”

Section: Related Workmentioning

confidence: 99%

“…However, previous work on crosslingual REE suffers from the monolingual bias issue due to the monolingual training of models on only the source language data, leading to non-optimal crosslingual performance. A solution for this issue can resort to language adversarial training Huang et al, 2019;Keung et al, 2019;Lange et al, 2020;He et al, 2020) where unlabeled data in the target language is used to aid crosslingual representations via fooling a language discriminator. The underlying principle for this approach is to encourage the closeness of representation vectors for sentences in the source and target languages (i.e., aligning representation vectors).…”

Section: Introductionmentioning

confidence: 99%

Crosslingual Transfer Learning for Relation and Event Extraction via Word Category and Class Alignments

Nguyen

Min³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Previous work on crosslingual Relation and Event Extraction (REE) suffers from the monolingual bias issue due to the training of models on only the source language data. An approach to overcome this issue is to use unlabeled data in the target language to aid the alignment of crosslingual representations, i.e., via fooling a language discriminator. However, as this approach does not condition on class information, a target language example of a class could be incorrectly aligned to a source language example of a different class. To address this issue, we propose a novel crosslingual alignment method that leverages class information of REE tasks for representation learning. In particular, we propose to learn two versions of representation vectors for each class in an REE task based on either source or target language examples. Representation vectors for corresponding classes will then be aligned to achieve class-aware alignment for crosslingual representations. In addition, we propose to further align representation vectors for languageuniversal word categories (i.e., parts of speech and dependency relations). As such, a novel filtering mechanism is presented to facilitate the learning of word category representations from contextualized representations on input texts based on adversarial learning. We conduct extensive crosslingual experiments with English, Chinese, and Arabic over REE tasks. The results demonstrate the benefits of the proposed method that significantly advances the state-of-the-art performance in these settings.

show abstract

“…(Lee et al, 2014;Zhong and Cambria, 2018). There are only a small number of fully supervised approaches in this field (Laparra et al, 2018;Lange et al, 2020), and despite the recent rise of transformer-based language models and their ability to generalize well, only the subtask of temporal tagging has been approached with this sort of architecture . One limiting factor for an end-to-end supervised model is the amount of labeled data it requires, which is not necessarily satisfied by the available resources.…”

Section: Introductionmentioning

confidence: 99%

BERT got a Date: Introducing Transformers to Temporal Tagging

Almasian¹,

Aumiller²,

Gertz³

2021

Preprint

View full text Add to dashboard Cite

Temporal expressions in text play a significant role in language understanding and correctly identifying them is fundamental to various retrieval and natural language processing systems. Previous works have slowly shifted from rule-based to neural architectures, capable of tagging expressions with higher accuracy. However, neural models can not yet distinguish between different expression types at the same level as their rule-based counterparts. In this work, we aim to identify the most suitable transformer architecture for joint temporal tagging and type classification, as well as, investigating the effect of semi-supervised training on the performance of these systems. Based on our study of token classification variants and encoder-decoder architectures, we present a transformer encoder-decoder model using the RoBERTa language model as our best performing system. By supplementing training resources with weakly labeled data from rule-based systems, our model surpasses previous works in temporal tagging and type classification, especially on rare classes. Our code and pre-trained experiments are available at: https://github.com/satya77/ Transformer_Temporal_Tagger

show abstract

Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text

Cited by 14 publications

References 28 publications

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

Crosslingual Transfer Learning for Relation and Event Extraction via Word Category and Class Alignments

BERT got a Date: Introducing Transformers to Temporal Tagging

Contact Info

Product

Resources

About