Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling

Mochihashi, Daichi; Yamada, Takeshi; Ueda, Naonori

doi:10.3115/1687878.1687894

Cited by 171 publications

(226 citation statements)

References 16 publications

Supporting

Mentioning

225

Contrasting

Unclassified

Order By: Relevance

“…Mochihashi's method can automatically detect words from character strings in every language without a prepared dictionary (Mochihashi et al, 2009). Mochihashi's algorithm can automatically perform word separation in a sentence including spoken words, abbreviations, and new words for all kinds of languages, whereas previous algorithms were not able to deal with such processing.…”

Section: Mochihashi's Methodsmentioning

confidence: 99%

Language-Independent Word Acquisition Method Using a State-Transition Model

Yamagishi

Suzuki

et al. 2016

Industrial Engineering and Management Systems

View full text Add to dashboard Cite

The use of new words, numerous spoken languages, and abbreviations on the Internet is extensive. As such, automatically acquiring words for the purpose of analyzing Internet content is very difficult. In a previous study, we proposed a method for Japanese word segmentation using character N-grams. The previously proposed method is based on a simple state-transition model that is established under the assumption that the input document is described based on four states (denoted as A, B, C, and D) specified beforehand: state A represents words (nouns, verbs, etc.); state B represents statement separators (punctuation marks, conjunctions, etc.); state C represents postpositions (namely, words that follow nouns); and state D represents prepositions (namely, words that precede nouns). According to this state-transition model, based on the states applied to each pseudo-word, we search the document from beginning to end for an accessible pattern. In other words, the process of this transition detects some words during the search. In the present paper, we perform experiments based on the proposed word acquisition algorithm using Japanese and Chinese newspaper articles. These articles were obtained from Japan's Kyoto University and the Chinese People's Daily. The proposed method does not depend on the language structure. If text documents are expressed in Unicode the proposed method can, using the same algorithm, obtain words in Japanese and Chinese, which do not contain spaces between words. Hence, we demonstrate that the proposed method is language independent.

show abstract

Section: Mochihashi's Methodsmentioning

confidence: 99%

Language-Independent Word Acquisition Method Using a State-Transition Model

Yamagishi

Suzuki

et al. 2016

Industrial Engineering and Management Systems

View full text Add to dashboard Cite

show abstract

“…The sampling procedure is based on dynamic programming. More details of sampling procedure can be found in (Mochihashi et al, 2009). …”

Section: Nested Pitman-yor Processmentioning

confidence: 99%

“…Nested Pitman-Yor process is an extension of above described process, used to produce word segmentation of languages (Mochihashi et al, 2009) and creation of language models for speech recognition (Mousa et al, 2013). The difference between basic and nested Pitman-Yor process models is that the base measure G 0 is replaced by a another Pitman-Yor process of syllable n-grams.…”

Section: Nested Pitman-yor Processmentioning

confidence: 99%

Unsupervised learning of agglutinated morphology using nested Pitman-Yor process based morpheme induction algorithm

Кумар

Padró

Oliver

2015

2015 International Conference on Asian Language Processing (IALP)

View full text Add to dashboard Cite

In this paper we describe a method to morphologically segment highly agglutinating and inflectional languages from Dravidian family. We use nested Pitman-Yor process to segment long agglutinated words into their basic components, and use a corpus based morpheme induction algorithm to perform morpheme segmentation. We test our method in two languages, Malayalam and Kannada and compare the results with Morfessor.

show abstract

“…In (Goldwater et al, 2006) they report issues with mixing in the sampler that were overcome using annealing. In (Mochihashi et al, 2009) this issue was overcome by using a blocked sampler together with a dynamic programming approach. Our algorithm is an extension of application the forward filtering backward sampling (FFBS) algorithm (Scott, 2002) to the problem of word segmentation presented in (Mochihashi et al, 2009).…”

Section: Bayesian Inferencementioning

confidence: 99%

“…These methods can be roughly classified into dictionary-based (Sornlertlamvanich, 1993;Srithirath and Seresangtakul, 2013) and statistical methods (Wu and Tseng, 1993;Maosong et al, 1998;Papageorgiou and P., 1994;Mochihashi et al, 2009;Jyun-Shen et al, 1991). In dictionary-based methods, only words that are stored in the dictionary can be identified and the performance depends to a large degree upon the coverage of the dictionary.…”

Section: Related Workmentioning

confidence: 99%

Integrating Dictionaries into an Unsupervised Model for Myanmar Word Segmentation

Thu

Finch

Sumita

et al. 2014

Proceedings of the Fifth Workshop on South and Southeast Asian Natural Language Processing

View full text Add to dashboard Cite

This paper addresses the problem of word segmentation for low resource languages, with the main focus being on Myanmar language. In our proposed method, we focus on exploiting limited amounts of dictionary resource, in an attempt to improve the segmentation quality of an unsupervised word segmenter. Three models are proposed. In the first, a set of dictionaries (separate dictionaries for different classes of words) are directly introduced into the generative model. In the second, a language model was built from the dictionaries, and the n-gram model was inserted into the generative model. This model was expected to model words that did not occur in the training data. The third model was a combination of the previous two models. We evaluated our approach on a corpus of manually annotated data. Our results show that the proposed methods are able to improve over a fully unsupervised baseline system. The best of our systems improved the F-score from 0.48 to 0.66. In addition to segmenting the data, one proposed method is also able to partially label the segmented corpus with POS tags. We found that these labels were approximately 66% accurate.

show abstract

Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling

Cited by 171 publications

References 16 publications

Language-Independent Word Acquisition Method Using a State-Transition Model

Language-Independent Word Acquisition Method Using a State-Transition Model

Unsupervised learning of agglutinated morphology using nested Pitman-Yor process based morpheme induction algorithm

Integrating Dictionaries into an Unsupervised Model for Myanmar Word Segmentation

Contact Info

Product

Resources

About