Splitting-merging model of Chinese word tokenization and segmentation

Abstract

Currently, word tokenization and segmentation are still a hot topic in natural language processing, especially for languages like Chinese in which there is no blank space for word delimitation. Three major problems are faced: (1) tokenizing direction and efficiency; (2) insufficient tokenization dictionary and new words; and (3) ambiguity of tokenization and segmentation. Most existing tokenization and segmentation methods have not dealt with the above problems together. To tackle the three problems in one basket, this paper presents a novel dictionary-based method called the Splitting-Merging Model (SMM) for Chinese word tokenization and segmentation. It uses the mutual information of Chinese characters to find the boundaries and the non-boundaries of Chinese words, and finally leads to a word segmentation by resolving ambiguities and detecting new words.

Using Semi-Automatically Annotation System on Medical Entity Recognition

“…Unlike Latin based or Germanic based languages, the ones without blank space will lead to problems on word delimitation [21]. Major problems faced during tokenization and segmentation include tokenizing direction, e ciency, and ambiguity [22] [23]. CKIP word segmentation system was developed to resolve these problems.…”

The head-modifier principle and multilingual term extraction

“…A prerequisite to information extraction that is peculiar to Chinese language texts is a fundamental pre-processing task, namely word segmentation since Chinese natural language texts do not encode word boundaries. Approaches to segmentation have been both symbolic (rule-based), for example, Yeh and Lee (1991), and statistical, for example, Chen and Liu (1992), Yao and Lua (1998) and Peng (2001). Apart from this, a major focus of Chinese IE has been the recognition and classification of named entities, a task motivated by the significantly high distribution of proper nouns in newspaper texts.…”