Combination and boundary detection approaches on Chinese indexing

Abstract

Digital libraries store materials in electronic format. Research and development in digital libraries includes content creation, conversion, indexing, organization, and dissemination. The key technological issues are how to search and display desired selections from and across large collections effectively [Schatz & Chen, 1996]. Digital library research projects (DLI-1) sponsored by NSF/ DARPA/NASA have a common theme of bringing search to the net, which is the flagship research effort for the National Information Infrastructure (NII) in the United States. A repository is an indexed collection of objects.Indexing is an important task for searching. The better the indexing, the better the searching result. Developing a universal digital library has been the dream of many researchers, however, there are still many problems to be solved before such a vision is fulfilled. The most critical is to support a cross-lingual retrieval or multilingual digital library. Much work has been done on English information retrieval, however, there is relatively less work on Chinese information retrieval. In this article, we focus on Chinese indexing, which is the foundation of Chinese and cross-lingual information retrieval. The smallest indexing units in Chinese digital libraries are words, while the smallest units in a Chinese sentence are characters. However, Chinese text has no delimiter to mark word boundaries as it is in English text. In English or other languages using Roman or Greek-based orthographies, often, spacing reliably indicates word boundaries. In Chinese, a number of characters are placed together without any delimiters indicating the boundaries between consecutive characters. In this article, we investigate the combination and boundary detection approaches based on mutual information for segmentation. The combination approach combines ngrams to form words with more number of characters. In the combination approach Algorithm 1 does not allow overlapping of n-grams while Algorithm 2 does. The boundary detection approach detects the segmentation points on a sentence based on the values and the change of values of the mutual information. Experiments are conducted to evaluate their performances.

The research and application of the Chinese machinery word segmentation algorithm based on improved PATRICIA tree dictionary

“…Add a prefix tag judgments, and the maximum matching algorithm as being a dynamic value to return (3). The principle of priority based on long term, the maximum term and a maximum separation of the word, better solve the ambiguity problem, make up the largest sub-word segmentation algorithm and the algorithm.…”

“…Check dictionary, and it returns whether the word is the biggest word. If Is the word the word, the word is not the biggest offset offset +1, return to (3). Is the largest word is the starting position start +1, return to (2); (5).…”

“…Since the 90's, it has continued to raise many new segmentation algorithms [3]. These algorithms are beginning to base dictionary mechanisms, and have better space complexity.…”

Current research issues and trends in non-English Web searching

“…Word boundary identification is even harder in Asian languages such as Chinese (Chen and Liu 1992;Yang et al 2000;Foo and Li 2004), for example, since words are not delimited by blanks. Foo and Li (2004) conducted experiments to study the impact of Chinese word segmentation and its effect on IR.…”

Error anaylsis of Chinese text segmentation using statistical approach

“…Two statistical based Chinese text segmentation techniques have been developed by Yang et al, namely, boundary detection [3] and heuristic method [4]. Due to the limitation of the lexical statistics collected from the Chinese corpus, errors may occur in segmentation.…”