Content Selection and Curation for Web Archiving

Abstract

Any preservation effort must begin with an assessment of what content to preserve, and web archiving is no different. There have historically been two answers to the question "what should we archive?" The Internet Archive's broad entire-web crawls have been supplemented by narrower domain-or topic-specific collections gathered by numerous libraries. We can characterize this as content selection and curation by "gatekeepers". In contrast, we have witnessed the emergence of another approach driven by "the masses"-we can archive pages that are contained in social media streams such as Twitter. The interesting question, of course, is how these approaches differ. We provide an answer to this question in the context of a case study about the 2015 Canadian federal elections. Based on our analysis, we recommend a hybrid approach that combines an effort driven by social media and more traditional curatorial methods.

Observing Web Archives

“…Recent qualitative research on web archival appraisal practices by Summers and Punzalan [70] underlines the value of such an approach for both situating web archiving within wider institutional/archival paradigms, as well as exposing undocumented practices largely missing from the archival record. Taking a different approach, Milligan et al [50] contribute to a discussion of curatorial practices by reverse engineering selection through a comparison of algorithmic, manual and social media-generated web archives associated with the 2015 Canadian Elections. These studies, plus the previously mentioned work of Dougherty and Meyer [21], can be seen as complementary to this research both in methodology and in their aims to address how collection practices structure the nature of web archival engagement.…”

If these crawls could talk: Studying and documenting web archives provenance

“…Previous work begins to address these considerations of selection and scoping of a crawl, and how decisions are then reflected in the resulting web archives collection. Milligan, Ruest, and Lin () compare two collections with varying selection processes, calling for more self‐reflective practice and consideration of these choices. Emerging research focuses on the situated practice of web archiving and the activities that are involved in web crawling and constructing a collection.…”

What's cached is prologue: Reviewing recent web archives research towards supporting scholarly use

“…Samar et al (2016) analyze coverage of trending topics for the Netherlands in 2014 by comparing the National Library of the Netherlands' web archive to the Common Crawl dataset. Milligan et al (2016) use a case study of Canadian federal elections, comparing collections resulting from three different crawling strategies: starting with curated seed lists, collecting URLs from public tweets for a given hashtag, and broad crawls by the Internet Archive. Hale et al (2017) similarly compare the archive and the live web for a population of web pages from TripAdvisor.…”