Morphological Lexicon Extraction from Raw Text Data

Forsberg, Markus; Hammarström, Harald; Ranta, Aarne

doi:10.1007/11816508_49

Cited by 16 publications

(18 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…base forms with paradigm information. Forsberg et al (2006) used an approach that automatically deduces extraction rules for which they could nd as much support as was logically possible in order to make a safe inference. This leads to rules safely extracting words that already have a number of word forms in the corpus, i.e.…”

Section: Comparison With Results From Similar or Related Effortsmentioning

confidence: 99%

“…mid or high-frequency words. Such methods are especially suitable for resource-poor languages lacking readily available public domain morphological descriptions like the Ispell dictionaries (Kuenning, 2007) or similar . Forsberg et al (2006) concluded that it is recommendable that a linguist writes the extraction rules.…”

Section: Comparison With Results From Similar or Related Effortsmentioning

confidence: 99%

See 1 more Smart Citation

Entry Generation by Analogy – Encoding New Words for Morphological Lexicons

Lindén¹

2009

NEJLT

View full text Add to dashboard Cite

Language software applications encounter new words, e.g., acronyms, technical terminology, loan words, names or compounds of such words. To add new words to a lexicon, we need to indicate their base form and inflectional paradigm. In this article, we evaluate a combination of corpus-based and lexicon-based methods for assigning the base form and inflectional paradigm to new words in Finnish, Swedish and English finite-state transducer lexicons. The methods have been implemented with the open-source Helsinki Finite-State Technology (Lindén & al., 2009). As an entry generator often produces numerous suggestions, it is important that the best suggestions be among the first few, otherwise it may become more efficient to create the entries by hand. By combining the probabilities calculated from corpus data and from lexical data, we get a more precise combined model. The combined method has 77-81 % precision and 89-97 % recall, i.e. the first correctly generated entry is on the average found as the first or second candidate for the test languages. A further study demonstrated that a native speaker could revise suggestions from the entry generator at a speed of 300-400 entries per hour.

show abstract

Section: Comparison With Results From Similar or Related Effortsmentioning

confidence: 99%

Section: Comparison With Results From Similar or Related Effortsmentioning

confidence: 99%

Entry Generation by Analogy – Encoding New Words for Morphological Lexicons

Lindén¹

2009

NEJLT

View full text Add to dashboard Cite

show abstract

“…Rather, we base our algorithm on extracting the longest common subsequence (LCS) shared by all forms in an inflection table, from which alignment of segments falls out naturally. Although our paradigm representation is similar to and inspired by that of Forsberg et al (2006) and Détrez and Ranta (2012), our method of generalizing from inflection tables to paradigms is novel.…”

Section: Previous Workmentioning

confidence: 99%

Semi-supervised learning of morphological paradigms and lexicons

Hulden¹,

Forsberg²,

Ahlberg³

2014

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We present a semi-supervised approach to the problem of paradigm induction from inflection tables. Our system extracts generalizations from inflection tables, representing the resulting paradigms in an abstract form. The process is intended to be language-independent, and to provide human-readable generalizations of paradigms. The tools we provide can be used by linguists for the rapid creation of lexical resources. We evaluate the system through an inflection table reconstruction task using Wiktionary data for German, Spanish, and Finnish. With no additional corpus information available, the evaluation yields per word form accuracy scores on inflecting unseen base forms in different languages ranging from 87.81% (German nouns) to 99.52% (Spanish verbs); with additional unlabeled text corpora available for training the scores range from 91.81% (German nouns) to 99.58% (Spanish verbs). We separately evaluate the system in a simulated task of Swedish lexicon creation, and show that on the basis of a small number of inflection tables, the system can accurately collect from a list of noun forms a lexicon with inflection information ranging from 100.0% correct (collect 100 words), to 96.4% correct (collect 1000 words).

show abstract

“…At one extreme, we find systems that are painstakingly and carefully designed by hand (Koskenniemi, 1983;Buckwalter, 2004;Habash and Rambow, 2006;Détrez and Ranta, 2012). Next on the continuum, we find work that focuses on defining morphological models with limited lexica that are then extended using raw text (Clément et al, 2004;Forsberg et al, 2006). In the middle of this continuum, we find efforts to learn complete paradigms using fully supervised methods relying on completely annotated data points with rich morphological information (Durrett and DeNero, 2013;Eskander et al, 2013).…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Morphology-Based Vocabulary Expansion

Rasooli

Lippincott

Habash

et al. 2014

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

We present a novel way of generating unseen words, which is useful for certain applications such as automatic speech recognition or optical character recognition in low-resource languages. We test our vocabulary generator on seven low-resource languages by measuring the decrease in out-of-vocabulary word rate on a held-out test set. The languages we study have very different morphological properties; we show how our results differ depending on the morphological complexity of the language. In our best result (on Assamese), our approach can predict 29% of the token-based out-of-vocabulary with a small amount of unlabeled training data.

show abstract

Morphological Lexicon Extraction from Raw Text Data

Cited by 16 publications

References 5 publications

Entry Generation by Analogy – Encoding New Words for Morphological Lexicons

Entry Generation by Analogy – Encoding New Words for Morphological Lexicons

Semi-supervised learning of morphological paradigms and lexicons

Unsupervised Morphology-Based Vocabulary Expansion

Contact Info

Product

Resources

About