Subword-based tagging by conditional random fields for Chinese word segmentation

Abstract

We proposed two approaches to improve Chinese word segmentation: a subword-based tagging and a confidence measure approach. We found the former achieved better performance than the existing character-based tagging, and the latter improved segmentation further by combining the former with a dictionary-based segmentation. In addition, the latter can be used to balance out-of-vocabulary rates and in-vocabulary rates. By these techniques we achieved higher F-scores in CITYU, PKU and MSR corpora than the best results from Sighan Bakeoff 2005.

Max-Margin Tensor Neural Network for Chinese Word Segmentation

“…5 https://code.google.com/p/word2vec/ Model PKU MSRA Best05 95.0 96.0 Best05 (Tseng et al, 2005) 95.0 96.4 (Zhang et al, 2006) 95.1 97.1 (Zhang and Clark, 2007) 94.5 97.2 95.2 97.3 (Sun et al, 2012) 95.4 97.4 (Zhang et al, 2013) 96 A very common feature in Chinese word segmentation is the character bigram feature. Formally, at the i-th character of a sentence c [1:n] , the bigram features are c k c k+1 (i − 3 < k < i + 2).…”

Two Knives Cut Better Than One: Chinese Word Segmentation with Dual Decomposition

“…There are two primary classes of models: character-based, where the foundational units for processing are individual Chinese characters (Xue, 2003;Tseng et al, 2005;Zhang et al, 2006;Wang et al, 2010), and word-based, where the units are full words based on some dictionary or training lexicon (Andrew, 2006;Zhang and Clark, 2007). Sun (2010) details their respective theoretical strengths: character-based approaches better model the internal compositional structure of words and are therefore more effective at inducing new OOV words; word-based approaches are better at reproducing the words of the training lexicon and can capture information from significantly larger contextual spans.…”

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

“…Another approach has been that of Cherry and Guo (2015) and Peng and Dredze (2015), who relied on training unsupervised lexical embeddings in place of these upstream systems and achieved state-of-the-art results for English and Chinese social media, respectively. The same approach was also found helpful for NER in the news domain (Collobert and Weston, 2008;Passos et al, 2014) In Asian languages like Chinese, Japanese and Korean, word segmentation is a critical first step for many tasks (Gao et al, 2005;Zhang et al, 2006;Mao et al, 2008). Peng and Dredze (2015) showed the value of word segmentation to Chinese NER in social media by using character positional embeddings, which encoded word segmentation information.…”