Plain Text & Character Encoding: A Primer for Data Curators

Abstract

Plain text data consists of a sequence of encoded characters or “code points” from a given standard such as the Unicode Standard. Some of the most common file formats for digital data used in eScience (CSV, XML, and JSON, for example) are built atop plain text standards. Plain text representations of digital data are often preferred because plain text formats are relatively stable, and they facilitate reuse and interoperability. Despite its ubiquity, plain text is not as plain as it may seem. The set of standards used in modern text encoding (principally, the Unicode Character Set and the related encoding format, UTF-8) have complex architectures when compared to historical standards like ASCII. Further, while the Unicode standard has gained in prominence, text encoding problems are not uncommon in research data curation. This primer provides conceptual foundations for modern text encoding and guidance for common curation and preservation actions related to textual data.

Replicating Digital Audiovisual Archives in the Cloud

“…Most of the writings I reviewed in preparation for this report focus on the basics: introducing key standards (e.g., ASCII, Unicode) and the complications of identifying and converting plain text formats. Plain Text & Character Encoding: A Primer for Data Curators by Seth Erickson (2021) provides an excellent introduction to character encoding as it relates to digital preservation. Dig a little deeper and one will inevitably find more ethically and culturally-grounded conversations surrounding the implications of diacritic handling, examining the "racial and Western-centric implications of considering certain characters to be 'illegal'" (Blewer, 2019).…”