A Two-Level Structure for Compressing Aligned Bitexts

Abstract

Abstract.A bitext, or bilingual parallel corpus, consists of two texts, each one in a different language, that are mutual translations. Bitexts are very useful in linguistic engineering because they are used as source of knowledge for different purposes. In this paper we propose a strategy to efficiently compress and use bitexts, saving, not only space, but also processing time when exploiting them. Our strategy is based on a two-level structure for the vocabularies, and on the use of biwords, a pair of associated words, one from each language, as basic symbols to be encoded with an ETDC [2] compressor. The resulting compressed bitext needs around 20% of the space and allows more efficient implementations of the different types of searches and operations that linguistic engineerings need to perform on them. In this paper we discuss and provide results for compression, decompression, different types of searches, and bilingual snippets extraction.

Humans Outperform Machines at the Bilingual Shannon Game

“…The Hutter Prize [12], a competition to compress a 100 m-word extract of English Wikipedia, was designed to futher encourage research in text compression. Bilingual and multilingual text compression is a less-studied field [1,[13][14][15][16][17][18]. These papers provide different algorithms for compressing text in multilingual format, but they do not demonstrate how humans perform on this task.…”

Improved Alignment-Based Algorithm for Multilingual Text Compression

“…Another quite realistic assumption is that a source sequence already referred to once during the process will be rarely used again. The search for the next unreferenced sequence is always very local, meaning that it is performed in time O (1).…”

“…The idea of using alignment was raised in [6], and a detailed algorithm was presented in [7]. Different, though basically similar algorithms have recently been suggested in [13,1]. However, all these algorithms relate only to bilingual parallel texts; therefore, should three or more parallel texts be compressed, the algorithms would be applied to each sourcetarget pair of texts independently.…”

Boosting Bitext Compression

“…A biword-based scheme, called 2lcab, is proposed in [1]. 2lcab builds a two-level dictionary in which word and biword representations are stored.…”

“…Table 2 shows compression and decompression times, respectively; word alignment times are note taken into account. Due to lack of space we only show 2lcab and trc 1 as boosters times, however all the times obtained by using 1v and 2v as boosters are similar to those of trc 1 . Results show that the techniques we propose are very fast at compression and, mainly, at decompression.…”