Chinese Unknown Word Recognition Using Improved Conditional Random Fields

Xu, Yisu; Wang, Xuan; Tang, Buzhou; Wang, Xiaolong

doi:10.1109/isda.2008.283

Cited by 3 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because it is difficult for these methods to choose the appropriate threshold, some researchers try to combine machine learning strategies, such as support vector machine (SVM) [15][16][17][18], conditional random fields (CRF) [19,20], hidden Markov model (HMM) [21][22][23] and so on. Peng et al [8] and Xu et al [24] used the model of CRF. Peng et al considered word segmentation and new word extraction as a unified process.…”

Section: Related Workmentioning

confidence: 99%

A novel unsupervised method for new word extraction

Lili

Huang

Wei

et al. 2016

Sci. China Inf. Sci.

View full text Add to dashboard Cite

New words could benefit many NLP tasks such as sentence chunking and sentiment analysis. However, automatic new word extraction is a challenging task because new words usually have no fixed language pattern, and even appear with the new meanings of existing words. To tackle these problems, this paper proposes a novel method to extract new words. It not only considers domain specificity, but also combines with multiple statistical language knowledge. First, we perform a filtering algorithm to obtain a candidate list of new words. Then, we employ the statistical language knowledge to extract the top ranked new words. Experimental results show that our proposed method is able to extract a large number of new words both in Chinese and English corpus, and notably outperforms the state-of-the-art methods. Moreover, we also demonstrate our method increases the accuracy of Chinese word segmentation by 10% on corpus containing new words.

show abstract