FinnPos: an open-source morphological tagging and lemmatization toolkit for Finnish

Abstract

This paper describes FinnPos, an open-source morphological tagging and lemmatization toolkit for Finnish. The morphological tagging model is based on the averaged structured perceptron classifier. Given training data, new taggers are estimated in a computationally efficient manner using a combination of beam search and model cascade. The lemmatization is performed employing a combination of a rule-based morphological analyzer, OMorFi, and a data-driven lemmatization model. The toolkit is readily applicable for tagging and lemmatization of running text with models learned from the recently published Finnish Turku Dependency Treebank and FinnTreeBank. Empirical evaluation on these corpora shows that FinnPos performs favorably compared to reference systems in terms of tagging and lemmatization accuracy. In addition, we demonstrate that our system is highly competitive with regard to computational efficiency of learning new models and assigning analyses to novel sentences.

Data-Driven Spelling Correction using Weighted Finite-State Methods

“…The FinnPos tagger toolkit (Silfverberg et al, 2015) is used to train the models for the structured system and the HFST Python interface (Lindén et al, 2011) is used for constructing and operating finite-state machines. When training FinnPos models, we used default settings for most hyperparameters.…”

Using HFST—Helsinki Finite-State Technology for Recognizing Semantic Frames

“…FinnPos [20] is a data driven morphological tagging toolkit distributed with the HFST interface. The term morphological tagging [6] refers to assigning one full morphological label, including for example part-of-speech, tense, case and number, to each word in a text.…”

“…In contrast, FinnPos is especially geared toward morphologically rich languages with large label sets, that cause data sparsity and slow down estimation when using standard solutions. FinnPos gives state-of-the-art results for the morphologically rich language Finnish [20] both with regard to runtime and accuracy. In addition to morphological tagging, FinnPos also performs data driven lemmatization.…”

“…A more detailed description of the theoretical foundations as well as evaluation can be found in [20].…”

“…To speed up training, FinnPos uses an adaptive beam search and a label guesser [20] during inference and estimation. These substantially reduce run-time.…”