Efficient Search in Hidden Text of Large DjVu Documents

Abstract

The paper describes an open-source tool which allows to present endusers with results of advanced language technologies. It relies on the DjVu format, which for some applications is still superior to other modern formats including PDF/A. The DjVu GPLed tools are not limited just to the DjVuLibre library, but are being supplemented by various new programs, such as pdf2djvu developed by Jakub Wilk. It allows in particular to convert to DjVu the PDF output of popular OCR programs like FineReader preserving the hidden text layer and some other features.The tool in question has been conceived by the present author and consist of a modification of the Poliqarp corpus query tool, used for National Corpus of Polish; his ideas have been very succesfully implemented by Jakub Wilk. The new system, called here simply Poliqarp for DjVu, inherits from its origin not only the powerfull search facilities based on two-level regular expressions, but also the ability to represent low-level ambiguities and other linguistic phenomena. Although at present the tool is used mainly to facilitate access to the results of dirty OCR, it is ready to handle also more sophisticated output of linguistic technologies.

The IMPACT project Polish Ground-Truth texts as a Djvu corpus

“…It was designed by the present author and implemented by Jakub Wilk. The reasons for selecting the DjVu format were presented on several occasions, such as (Bień, 2009) and (Bień, 2011). In this paper we focus on problems specific to the IMPACT data.…”

Korpusomat – a Tool for Creating Searchable Morphosyntactically Tagged Corpora

“…On the contrary, Poliqarp is currently no longer updated but, at the same time it is still under use in several projects [13][14][15].…”