Enhancing deep neural networks with morphological information

Abstract

Deep learning approaches are superior in natural language processing due to their ability to extract informative features and patterns from languages. The two most successful neural architectures are LSTM and transformers, used in large pretrained language models such as BERT. While cross-lingual approaches are on the rise, most current natural language processing techniques are designed and applied to English, and less-resourced languages are lagging behind. In morphologically rich languages, information is conveyed through morphology, for example, through affixes modifying stems of words. The existing neural approaches do not explicitly use the information on word morphology. We analyse the effect of adding morphological features to LSTM and BERT models. As a testbed, we use three tasks available in many less-resourced languages: named entity recognition (NER), dependency parsing (DP) and comment filtering (CF). We construct baselines involving LSTM and BERT models, which we adjust by adding additional input in the form of part of speech (POS) tags and universal features. We compare the models across several languages from different language families. Our results suggest that adding morphological features has mixed effects depending on the quality of features and the task. The features improve the performance of LSTM-based models on the NER and DP tasks, while they do not benefit the performance on the CF task. For BERT-based models, the added morphological features only improve the performance on DP when they are of high quality (i.e., manually checked) while not showing any practical improvement when they are predicted. Even for high-quality features, the improvements are less pronounced in language-specific BERT variants compared to massively multilingual BERT models. As in NER and CF datasets manually checked features are not available, we only experiment with predicted features and find that they do not cause any practical improvement in performance.

Proceedings of the Second Workshop on NLP Applications to Field Linguistics

“…Morphological feature embeddings Adding morphological features explicitly as input on NLP tasks has mixed effects, depending on the task and quality of features. Klemen et al (2022) show across several languages that the results on (monolingual) dependency parsing and named entity recognition improve on LSTM-based models when UD feature embeddings are added as input, while the performance on comment filtering is not affected. Manually annotated features yield better results than automatically added features.…”

Attention mechanism and skip-gram embedded phrases

“…Mining n-grams is the automatic extraction of frequent phrases (Del, Tättar, and Fishel 2018), such as multi-word terms and special phrases, from a corpus. First, we POS tag, parse (syntactic dependencies) (Klemen, Krsnik, and Robnik-Šikonja 2022), lemmatise and tokenise the whole corpus and then extract bigrams, trigrams and tetragrams, hereinafter referred to as n-grams (verbs, nouns, adverbs, adjectives, participles and prepositions) to subsequently take them as input into Word2Vec, in particular into the Skip-gram algorithm, which generates vectorised words of high dimensionality (Camacho-Collados and Pilehvar 2018) with more meaning (see Figure 9). The threshold for the n-grams will be high, so that high quality legal LSP words (especially with short-and longdistance dependencies), phrases are extracted.…”

Analysis of the Theoretical Foundations of Neural Network Modeling of Language Unit Recognition

“…The effectiveness of Deep Learning in natural language processing (NLP) was studied in the work of Klemen M., Krsnik L., Robnik-Šikonja M. [30]. In their research, the authors argue that the use of Deep Learning in natural language processing (NLP) is more productive than other methods.…”