Creating order from the mess: web archive derivative datasets and notebooks

All WARC and no playback: The materialities of data-centered web archives research

“…Together these examples demonstrate the significant work required—conceptual re-framing as well as human and technical resources—to re-order WARC data to accommodate researchers’ information needs. I contrast my findings with recent work that positions the WARC as an “unaccessioned collection,” analogous to a “shipping container full of banker boxes holding dozens of jumbled records” which has not yet been processed by archivists (Ruest et al, 2022: 2). I argue here that WARC data is in fact an organized set of records, but ordered according to the logic of the crawler rather than the logics of human users or curators.…”

“…I examine here how the WARC file format has come to play an essential role in web archiving, and the actors driving its development as a standard. While prior work briefly addresses the WARC format's structure and resulting challenges for research (Ruest et al, 2022), my investigation here traces both the format's origins and undertakes a close reading of its design. By drawing parallels to the case of the MVZ, I identify the WARC's role in standardizing collection methods for the web archiving community.…”

“…It is premised on the fact that data-centered “whole collection” analyses of web archives rely upon standardized data artifacts. While past work has identified the use of data derivatives and outlined methodological approaches (Weber, 2018; Ruest et al, 2022), I focus on the construction of data artifacts themselves as a determining factor that ultimately shapes and constrains research. The materiality of archived objects and web pages (relative to live-web counterparts) has been a longstanding topic of interest for web archives scholars (see, Brügger, 2013; Helmond, 2017; Rogers, 2017), however the materiality of archived data remains an undertheorized area.…”

“…In this context, I study how WARC data are used by teams of researchers through ethnographic fieldwork in two settings. While prior work addresses the WARC format's structure and resulting challenges for research (Ruest et al, 2022), I take an in-depth approach pairing fieldwork with an investigation of the format's origins, and close reading of its design. My work is motivated by a need to understand how researchers studying archived web data negotiate the particularities of data materiality, and how representational forms ultimately shape and constrain research practices.…”

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

“…Some organizations like the Library of Congress [9] and the UK Web Archive [15] have found a way to offer more access to web archives without providing the primary datasets. These organizations offer access to derivative datasets, which are made of sampled data taken from a web archive [12]. Derivative datasets can include things like a collection of PDF files from a web archive, a Geoindex, or crawled URL indexes.…”