Overview of Chinese Word Segmentation Method

Abstract

This article provides a brief introduction to Natural Language Processing and basic knowledge of Chinese Word Segmentation at first. Chinese Word Segmentation is a process of turning a series of Chinese characters into a series of Chinese words with some rules. As the fundamental component of Chinese information processing, it is wildly used in correlative areas. Accordingly, research on Chinese Word Segmentation has important theoretic and realistic meaning. In this paper, we mainly introduces the challenge in Chinese Word Segmentation, and presented the categories of Chinese Word Segmentation method.

Research on Words Segmentation Technology in Chinese Full Text Retrieval System

“…In current, some achievement has been achieved in the research and application of Chinese word s egmentation, and some effective algorithms have been proposed. There are three main categories, di ctionary base word segmentation algorithm [4] (Forward maximum matching algorithm, Reverse ma ximum matching algorithm), statistics based word segmentation algorithm [5] (Mutual Information p robabilistic algorithms, Decision algorithm combined degree) and rules based word segmentation me thod [6]. The above algorithms laid the foundation of full-text retrieval technology, but the applicatio n demonstrates that each method has some limitations in ambiguity dealing, word length limitation, and time cost.…”

Identification and Analysis of Multi-tasking Product Information Search Sessions with Query Logs

“…We use Rwordseg (Li, 2013) as the default dictionary and an additional dictionary containing terms from the Product Catalog acquired from Taobao API  for query term segmentation. Then we calculate the pairwise Jaccard index (Järvelin et al, 2007) of queries that belong to a same user, and construct a similarity matrix based on the Jaccard values.…”