Applying Occam’s Razor to Transformer-Based Dependency Parsing: What Works, What Doesn’t, and What is Really Necessary

Grünewald, Stefan; Friedrich, Annemarie; Kuhn, Jonas

doi:10.18653/v1/2021.iwpt-1.13

Cited by 5 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ILSP Neural NLP Toolkit for Greek (Prokopidis & Piperidis, 2020) HR (Ljubešić & Dobrovoljc, 2019;Terčon & Ljubešić, 2023) HU ▲ •• huspacy (Orosz, Szántó, Berkecz, Szabó, & Farkas, 2022) IS Stanza (Qi et al, 2020) LV (Ljubešić & Dobrovoljc, 2019;Terčon & Ljubešić, 2023) TR Çöltekin, 2010), ▲ steps-parser (Grünewald, Friedrich, & Kuhn, 2021), TurkishNER UA…”

Section: Compiling Individual Corporamentioning

confidence: 99%

ParlaMint II: Advancing Comparable Parliamentary Corpora Across Europe

Erjavec,

Kopp,

Ljubešić

et al. 2024

Preprint

View full text Add to dashboard Cite

The paper presents the results of the ParlaMint II project, which comprise comparable corpora of parliamentary debates of 29 European countries and autonomous regions, covering at least the period from 2015 to 2022, and containing over 1 billion words. The corpora are uniformly encoded, contain rich metadata about their 24 thousand speakers, and are linguistically annotated up to the level of Universal Dependencies syntax and named entities. The paper focuses on the enhancement made since the ParlaMint I project and presents the compilation of the corpora, including the encoding infrastructure, use of GitHub, the production of individual corpora, the common pipeline for producing their distribution, and use of CLARIN services for dissemination. It then gives a quantitative overview of the produced corpora, followed by the qualitative additions made within the ParlaMint II project, namely metadata localisation, the addition of new metadata, such as the political orientation of political parties, the machine translation of the corpora to English and its tagging with semantic classes, and the production of pilot speech corpora. Finally, outreach activities and further work are discussed.

show abstract

Section: Compiling Individual Corporamentioning

confidence: 99%

ParlaMint II: Advancing Comparable Parliamentary Corpora Across Europe

Erjavec,

Kopp,

Ljubešić

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…So, we reimplement the auxiliary task modules and the combined parsing approach for an XLM-R-based encoding module. For this purpose we follow the XLM-R-based parsing architecture of Grünewald et al (2021) which has the same biaffine parsing model described in Dozat and Manning (2017). Our aim is to observe how extracting parsingrelated knowledge from semi-supervised auxiliary tasks affects a multilingual transformer model.…”

Section: Transformer-based Adaptation Of the Modelmentioning

confidence: 99%

Improving Code-Switching Dependency Parsing with Semi-Supervised Auxiliary Tasks

Özateş¹,

Özgür²,

Güngör³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Code-switching dependency parsing stands as a challenging task due to both the scarcity of necessary resources and the structural difficulties embedded in code-switched languages. In this study, we introduce novel sequence labeling models to be used as auxiliary tasks for dependency parsing of code-switched text in a semi-supervised scheme. We show that using auxiliary tasks enhances the performance of an LSTM-based dependency parsing model and leads to better results compared to an XLM-Rbased model with significantly less computational and space complexity. As the first study that focuses on multiple code-switching language pairs for dependency parsing, we acquire state-of-the-art scores on all of the studied languages. Our best models outperform the previous work by 7.4 LAS points on average.

show abstract

Enhancing deep neural networks with morphological information

2022

View full text Add to dashboard Cite

Deep learning approaches are superior in natural language processing due to their ability to extract informative features and patterns from languages. The two most successful neural architectures are LSTM and transformers, used in large pretrained language models such as BERT. While cross-lingual approaches are on the rise, most current natural language processing techniques are designed and applied to English, and less-resourced languages are lagging behind. In morphologically rich languages, information is conveyed through morphology, for example, through affixes modifying stems of words. The existing neural approaches do not explicitly use the information on word morphology. We analyse the effect of adding morphological features to LSTM and BERT models. As a testbed, we use three tasks available in many less-resourced languages: named entity recognition (NER), dependency parsing (DP) and comment filtering (CF). We construct baselines involving LSTM and BERT models, which we adjust by adding additional input in the form of part of speech (POS) tags and universal features. We compare the models across several languages from different language families. Our results suggest that adding morphological features has mixed effects depending on the quality of features and the task. The features improve the performance of LSTM-based models on the NER and DP tasks, while they do not benefit the performance on the CF task. For BERT-based models, the added morphological features only improve the performance on DP when they are of high quality (i.e., manually checked) while not showing any practical improvement when they are predicted. Even for high-quality features, the improvements are less pronounced in language-specific BERT variants compared to massively multilingual BERT models. As in NER and CF datasets manually checked features are not available, we only experiment with predicted features and find that they do not cause any practical improvement in performance.

show abstract

Applying Occam’s Razor to Transformer-Based Dependency Parsing: What Works, What Doesn’t, and What is Really Necessary

Cited by 5 publications

References 36 publications

ParlaMint II: Advancing Comparable Parliamentary Corpora Across Europe

ParlaMint II: Advancing Comparable Parliamentary Corpora Across Europe

Improving Code-Switching Dependency Parsing with Semi-Supervised Auxiliary Tasks

Enhancing deep neural networks with morphological information

Contact Info

Product

Resources

About