Context-based morphological disambiguation with random fields

Abstract

Finite-state approaches have been highly successful at describing the morphological processes of many languages. Such approaches have largely focused on modeling the phone-or character-level processes that generate candidate lexical types, rather than tokens in context. For the full analysis of words in context, disambiguation is also required (Hakkani-Tür et al., 2000;Hajič et al., 2001). In this paper, we apply a novel source-channel model to the problem of morphological disambiguation (segmentation into morphemes, lemmatization, and POS tagging) for concatenative, templatic, and inflectional languages. The channel model exploits an existing morphological dictionary, constraining each word's analysis to be linguistically valid. The source model is a factored, conditionally-estimated random field (Lafferty et al., 2001) that learns to disambiguate the full sentence by modeling local contexts. Compared with baseline state-of-the-art methods, our method achieves statistically significant error rate reductions on Korean, Arabic, and Czech, for various training set sizes and accuracy measures.

A comparison of segmentation methods and extended lexicon models for Arabic statistical machine translation

“…The method is also compared to an SVM based segmenter presented by Diab et al (2004) and shows improved results for small tasks but, again, no or only little improvement for large tasks. Nguyen and Vogel (2008) apply a Conditional Random Fields (CRF) segmentation method (as presented in Smith et al 2005) for Arabic-to-English translation. They show that a reduced morpheme segmentation, i.e.…”

Conditional Random Fields in Speech, Audio, and Language Processing

“…In subsequent work, Sutton et al [11] observe that jointly modeling noun phrase (NP) segmentation and POS tagging using DCRFs improves POS prediction accuracies. POS tagging using CRFs has also been applied to languages such as Chinese [77], Arabic, Czech [78], and Japanese [79], with complex morphological structures, and they have been found to outperform their generative counterparts.…”

Web-based frequency dictionaries for medium density languages

“…The standard solution to constrain the probabilistic tagging model for some of the unseen items is the application of MA (Hakkani-Tür et al, 2000;Hajič et al, 2001;Smith et al, 2005). Here a distinction must be made between those items that are not found in the training corpus (these we have called unseen tokens) and those that are not known to the MA -we call these out of vocabulary (OOV).…”