Subword-based tagging for confidence-dependent Chinese word segmentation

Abstract

We proposed a subword-based tagging for Chinese word segmentation to improve the existing character-based tagging. The subword-based tagging was implemented using the maximum entropy (MaxEnt) and the conditional random fields (CRF) methods. We found that the proposed subword-based tagging outperformed the character-based tagging in all comparative experiments. In addition, we proposed a confidence measure approach to combine the results of a dictionary-based and a subword-tagging-based segmentation. This approach can produce an ideal tradeoff between the in-vocaulary rate and out-of-vocabulary rate. Our techniques were evaluated using the test data from Sighan Bakeoff 2005. We achieved higher F-scores than the best results in three of the four corpora: PKU(0.951), CITYU(0.950) and MSR(0.971).

Utilizing the Web for Automatic Word Spacing

“…• CRF: the approach which uses CRF model, which has shown good performance in Chinese word segmentation tasks [10], [11]. We use linear-chain CRFs † and the standard BIO labels.…”

Chinese word segmentation as morpheme-based lexical chunking

“…Recent studies in CWS focus on tagging approaches with either characters [16,30] or words [4,33,34] as tagging units. Very little research [29] has been devoted to resolving CWS problems based on morphemes, a lower-level linguistic structure than words.…”

Multi-Grained Chinese Word Segmentation

“…To alleviate the high OOV-ratio issue of character-based sequence labeling, Zhang et al (2006) and Zhao and Kit (2007) propose subword-based sequence labeling for word segmentation by extracting highfrequency subword and treating them as the basic labeling units. Li (2011) and Li and Zhou (2012) propose to jointly parse the internal structures of words and syntactic structure of a sentence.…”