Generalized Biwords for Bitext Compression and Translation Spotting

Abstract

Large bilingual parallel texts (also known as bitexts) are usually stored in a compressed form, and previous work has shown that they can be more efficiently compressed if the fact that the two texts are mutual translations is exploited. For example, a bitext can be seen as a sequence of biwords ---pairs of parallel words with a high probability of co-occurrence--- that can be used as an intermediate representation in the compression process. However, the simple biword approach described in the literature can only exploit one-to-one word alignments and cannot tackle the reordering of words. We therefore introduce a generalization of biwords which can describe multi-word expressions and reorderings. We also describe some methods for the binary compression of generalized biword sequences, and compare their performance when different schemes are applied to the extraction of the biword sequence. In addition, we show that this generalization of biwords allows for the implementation of an efficient algorithm to look on the compressed bitext for words or text segments in one of the texts and retrieve their counterpart translations in the other text ---an application usually referred to as translation spotting--- with only some minor modifications in the compression algorithm

How Much Information Does a Human Translator Add to the Original?

“…Portions of this large corpus have been used in previous compression work (Sánchez-Martínez et al, 2012). The Spanish side is in UTF-8.…”

“…Sánchez-Martínez et al (2012) improve the interleaving scheme and include offsets to enable decompression to reconstruct the original word order. They also compare several characterbased and word-based compression schemes for biword sequences.…”

“…If we are predicting the jth English word, and we know that it translates f i ("aligns to f i "), and if f i has only a handful of translations, then we may be able to specify e j with just a few bits. We may therefore suppose that a set of Viterbi word alignments may be useful for compression (Conley and Klein, 2008;Sánchez-Martínez et al, 2012). We consider unidirectional alignments that link each target position j to a single source position i (including the null word at i = 0).…”

Humans Outperform Machines at the Bilingual Shannon Game

“…The Hutter Prize [12], a competition to compress a 100 m-word extract of English Wikipedia, was designed to futher encourage research in text compression. Bilingual and multilingual text compression is a less-studied field [1,[13][14][15][16][17][18]. These papers provide different algorithms for compressing text in multilingual format, but they do not demonstrate how humans perform on this task.…”

Phrasal Rank-Encoding: Exploiting Phrase Redundancy and Translational Relations for Phrase Table Compression

“…The most recent work is Sanchez-Martinez et al (2012) who propose to use "biwords" to compress parallel data sequentially. Similar as in Conley and Klein (2008), translational relations and Huffman coding are employed to take advantage of the improved entropy properties of the encoded data.…”