Injecting information into atomic units of text

Abstract

This paper presents a new approach to text processing, based on textemes. These are atomic text units generalising the concepts of character and glyph by merging them in a common data structure, together with an arbitrary number of user-defined properties. In the first part, we give a survey of the notions of character and glyph and their relation with Natural Language Processing models, some visual text representation issues and strategies adopted by file formats (SVG, PDF, DVI) and software (Uniscribe, Pango).In the second part we show applications of textemes in various text processing issues: ligatures, variant glyphs and other OpenType-related properties, hyphenation, color and other presentation attributes, Arabic form and morphology, CJK spacing, metadata, etc. Finally we describe how the Omega typesetting system implements texteme processing as an example of a generalised approach to input character stream parsing, internal representation of text, and modular typographic transformations. In the data flow from input to output, whether in memory or through serializations in auxiliary data files, textemes progressively accumulate information that is used by Omega's paragraph builder engine and included in the output DVI file. We show how this additional information increases efficiency of conversions to other file formats such as PDF or SVG. We conclude this paper by presenting interesting potential applications of texteme methods in document engineering.

WebKhoj

“…The computer has brought a more abstract layer to it, by storing and transmitting textual data. The atomic unit of this abstract representation of text, as defined in the Unicode standard [8], is called a character. And indeed, characters prove to be useful for obtaining alternative (nonvisual) representations of text such as Braille, speech synthesis, etc.…”

“…And indeed, characters prove to be useful for obtaining alternative (nonvisual) representations of text such as Braille, speech synthesis, etc. The visual representation of a character is called a glyph [8]. Displaying textual contents, whether on screen or on paper, involves translating characters into glyphs, a non-trivial operation for many writing systems.…”

“…Displaying textual contents, whether on screen or on paper, involves translating characters into glyphs, a non-trivial operation for many writing systems. Going in the opposite direction (from glyphs to characters) is known as OCR when done by a machine, or as reading when done by a human [8]. The technology trend over the last few years has been to use characters for most of the text processing and to limit glyph issues to the last stage, namely rendering.…”

Fontes intelligentes, textèmes et typographie dynamique

“…Notre but est de trouver une meilleure solution aux problèmes évoqués dans la section précédente -liés aux faiblesses du modèle d'Unicode et à la relation entre fontes intelligentes et paragraphage -mais aussi de mieux exploiter les informations fournies par les fontes intelligentes. Nous allons procéder par l'intermédiaire de trois nouveaux concepts : 1) le textème, introduit par les auteurs dans les articles (Haralambous et al, 2005a) et (Haralambous et al, 2005b) (sous le nom de signe dans ce dernier) ;…”

Research on rule definition and engine for general text processing

“…Some examples of these applications are text filtering, text retrieval, information extraction, information statistic, content rewriting, automatic generation of text, etc. However, a major drawback of the existing text processing solutions was that they develop the different processing programs to satisfy the corresponding text processing requirements [1][2][3].…”