A Japanese sentence analyzer

Maruyama, Newton; Morohashi, Masayuki; Umeda, S.; Sumita, Eiichro

doi:10.1147/rd.322.0238

Cited by 4 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Addressing this complexity, many algorithms for automated word segmentation exist [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18].…”

Section: Introductionmentioning

confidence: 99%

Automated Extraction of Lexicon Applied both to Chinese and Japanese Corpora

Shiu

Goertzel³

et al. 2012

2012 International Conference on Advanced Computer Science Applications and Technologies (ACSAT)

View full text Add to dashboard Cite

A novel statistical approach is described, enabling the automated extraction of large word lists from unsegmented corpora without reliance on existing dictionaries. The main contribution of this approach includes the following two points: First, it's very generic and has been successfully applied separately to both Chinese and Japanese; Second, it doesn't take any use of punctuation information, so compared to most of the existing methods, it doesn't need to pre-process the corpora to remove the punctuations or to pre-segment the corpora by punctuations. Our experiment results in the extraction of 14,087 Chinese words and 15,553 Japanese words. Precision achieved is over 80% for two-character Chinese words, over 90% for one-character Japanese words and over 70% for two-character Japanese words. And we ve also successfully extracted most of single-character words including common functional characters, such as " " (in), " " (and), " " (or), " "('s), " " (also), " " (a family name) in Chinese; hiragana such as " ,"" ,"" " in Japanese; and punctuations such as " ", " ", " ".

show abstract

“…Addressing this complexity, many algorithms for automated word segmentation exist [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18].…”

Section: Introductionmentioning

confidence: 99%