Extended models and tools for high-performance part-of-speech tagger

Abstract

Statistical part-of-st)eeeh(POS) taggers achieve high accuracy and robustness when based oil large, scale maimally tagged eorl)ora. Ilowever, enhancements of the learning models are necessary to achieve better 1)erforma.nce. We are develol)ing a learning tool for a Jalmnese morphological analyzer called Ch, aScn. Currently we use a fine-grained POS tag set with about 500 tags. To al)l)ly a normal trigram model on the tag set, we need unrealistic size of eorl)ora. Even, for a hi-gram model, we ean-no~, 1)ret)are a llloderate size of an mmotated corpus, when we take all the tags as distinct. A usual technique to Col)e with such fine-grained tags is to reduce the size of the tag set 1)y grouping the set of tags into equivalence classes. We introduce the concept of position-wise 9rouping where the tag set is t)artitioned into dill'el'lint equivalence classes at each t)osition in the. conditional 1)rohabilities in the Markov Model. Moreover, to eoi)e with the data Sl)arsen(?ss prot)lem caused 1) 3, exceptional t)henon> ena, we introduce several other techniques such as word-level statistics, smoothing of word-level an(l P()S-level statistics and a selective tri-gram model. To help users determine probabilistic 1)arameters, we introduce an error-driven method for the pm'mneter selection. We then give results of exl)eriments to see the effect of the tools applied to an existing Jat)anese morphological analyzer.

Word and Comments Evaluation Using Recursive Evaluation in Lecture Questionnaire

“…This rank was determined by a one-sided Smirnov-Grubbs test (5% significance). (2-3) Comments were decomposed into morphemes using the ChaSen (9) , nouns, verbs, and adjectives were extracted. Then these were multiplied by the evaluation value for each rank of each word.…”

Can a Teleoperated Android Represent Personal Presence? - A Case Study With Children

“…Since all conversations were done in Japanese, we analyzed the transcripts to see how actively participants spoke in each task, which is similar to counting words in English sentences. Here, the transcripts were morphologically analyzed and split into tokens by the Chasen tokenizer (Asahara & Matsumoto, 2000). The numbers of extracted tokens were counted for each participant or entity for each task.…”

Support for Internet-Based Commonsense Processing – Causal Knowledge Discovery Using Japanese “If” Forms

“…For this purpose we extracted relation-oriented sentences for creating dictionaries as verb dictionary, noun dictionary and n-gram dictionaries using WWW corpus (1,907,086 sentences) retrieved with Larbin robot. The verbs and nouns dictionaries consist of 79,460 verbs and 134,189 nouns retrieved with help of ChaSen [8]. For creating scripts automatically, our system had to search for the relationships between verbs and nouns and also between verb pairs.…”