Automatic construction of English/Chinese parallel corpora

Abstract

As the demand for global information increases significantly, multilingual corpora has become a valuable linguistic resource for applications to cross-lingual information retrieval and natural language processing. In order to cross the boundaries that exist between different languages, dictionaries are the most typical tools. However, the general-purpose dictionary is less sensitive in both genre and domain. It is also impractical to manually construct tailored bilingual dictionaries or sophisticated multilingual thesauri for large applications. Corpusbased approaches, which do not have the limitation of dictionaries, provide a statistical translation model with which to cross the language boundary. There are many domain-specific parallel or comparable corpora that are employed in machine translation and cross-lingual information retrieval. Most of these are corpora between Indo-European languages, such as English/French and English/Spanish. The Asian/Indo-European corpus, especially English/Chinese corpus, is relatively sparse. The objective of the present research is to construct English/ Chinese parallel corpus automatically from the World Wide Web. In this paper, an alignment method is presented which is based on dynamic programming to identify the one-to-one Chinese and English title pairs. The method includes alignment at title level, word level and character level. The longest common subsequence (LCS) is applied to find the most reliable Chinese translation of an English word. As one word for a language may translate into two or more words repetitively in another language, the edit operation, deletion, is used to resolve redundancy. A score function is then proposed to determine the optimal title pairs. Experiments have been conducted to investigate the performance of the proposed method using the daily press release articles by the Hong Kong SAR government as the test bed. The precision of the result is 0.998 while the recall is 0.806.

Finding translations in scanned book collections

“…Much of this work has been done on either finding parallel sentences from small corpora [28] or web pages [23,26,28,32]. Most of the work on finding web page has utilized structural information -HTML markup such anchors, links, filenames -to find [23,26] parallel resources.…”

“…Alignment was specifically rejected as being too expensive. [32] limited the alignment to titles and a translation dictionary to find parallel texts. Much of the machine translation work seems to be on the extraction of bilingual dictionaries [11] rather than finding document translations in large corpora.…”

The impact analysis of language differences on an automatic multilingual text summarization system

“…In the case of Hong Kong, as a British colony for more than a century, there has been a bilingual culture. Consequently, the official languages are Chinese and English and many important documents are written in Chinese and English, using covert translation (Yang & Li, 2003). For example, most of the documents released by the government have both Chinese and English versions.…”

“…In earlier related studies, it was determined that grammatical and lexical differences do have a significant effect on text processing (Yang & Li, 2003). For example, a word in one language can be translated into one or more words in another language.…”

Exploiting the Web as the multilingual corpus for unknown query translation

“…Resnick (1999) addressed the issue of automatic language identification for acquiring parallel corpus of Web documents. Yang and Li (2003) tried to construct English-Chinese parallel corpora from the Web sites with monolingual subtree structure. Then, the parallel corpora could be used in automatic generation of English-Chinese thesaurus (Yang & Luk, 2003) as a promising tool for cross-lingual information retrieval.…”