Automatic Extraction of Lithuanian Cybersecurity Terms Using Deep Learning Approaches

Abstract

The paper presents the results of research on deep learning methods aiming to determine the most effective one for automatic extraction of Lithuanian terms from a specialized domain (cybersecurity) with very restricted resources. A semi-supervised approach to deep learning was chosen for the research as Lithuanian is a less resourced language and large amounts of data, necessary for unsupervised methods, are not available in the selected domain. The findings of the research show that Bi-LSTM network with Bidirectional Encoder Representations from Transformers (BERT) can achieve close to state-of-the-art results.

Transforming Term Extraction: Transformer-Based Approaches to Multilingual Term Extraction Across Domains

“…In their work, pretrained language models clearly outperformed a classification method based on a variety of features, such as statistical descriptors and the domain-specificity measure termhood (Kageura and Umino, 1996). A recently published approach (Rokas et al, 2020) relies on LSTM, GRU and BERT embeddings and achieves high F1 scores for ATE of Lithuanian terms in the cybersecurity domain. Several approaches build on word embed-dings to perform ATE on specific domains, such as medicine (e.g.…”

Methodological Framework for the Development of an English-Lithuanian Cybersecurity Termbase

“…The best results were achieved with Bidirectional Long Short-Term Memory model (Bi-LSTM) using multilingual Bidirectional Encoder Representations from Transformers (BERT) embeddings reaching F1 score of 78.6%. The achieved high score suggests that the semi-supervised deep learning approach is a way to go (Rokas et al, 2020).…”

“…Current terminology extraction methods employ machine learning and deep learning approaches. Our pilot study on automatic extraction of monolingual (Lithuanian) cybersecurity terms proved that this methodology allows achieving high results even with very limited resources (Rokas et al, 2020). In the pilot study, several setups of different neural network configurations were iteratively tested by comparing their results to the gold standard which was pre-trained on a very small manually annotated training data (66,706 words of which 1,258 cybersecurity terms were manually annotated) compiled specifically for extraction of cybersecurity terminology.…”

Building of Parallel and Comparable Cybersecurity Corpora for Bilingual Terminology Extraction

“…The project aims at employing current deep learning terminology extraction methods. In 2020, the project team (Rokas et al, 2020) completed a pilot study on semi-supervised automatic extraction of Lithuanian CS terms from a Lithuanian monolingual corpus. A small-scale manually annotated dataset (66,706 word corpus with 1,258 annotated cybersecurity terms) was used as a training data.…”

“…A small-scale manually annotated dataset (66,706 word corpus with 1,258 annotated cybersecurity terms) was used as a training data. The pilot study was performed in several stages: firstly, various baseline LSTM and GRU networks were tested using the Adam optimiser and FastText embeddings; secondly, each of the best baseline LSTM and GRU networks were tested with various optimisers; and finally, the best model was compared with a model that has been trained using multilingual BERT embeddings (Rokas et al, 2020). The latter approach proved to be the most efficient: Bidirectional Long Short-Term Memory model (Bi-LSTM) using multilingual Bidirectional Encoder Representations from Transformers (BERT) embeddings reached F1 score of 78.6%.…”