Profiling Web Archival Voids for Memento Routing

Alam, Sawood; Weigle, Michele C.; Nelson, Michael L.

doi:10.1109/jcdl52503.2021.00027

Cited by 6 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hashes of raw mementos are stored and made available via the Internet Archive's CDX API [68]. However, this hashing of the WARC Payload Digest is sensitive to content-encoding PLOS ONE [69]. That is, if the payload stored was received as compressed (e.g., GZip or Brotli [70]), then the payload digest would be different than if it were in plain text, even if the content served to the client at replay time would be the same (as any stored content-encoding is undone at replay time).…”

Section: Verifying the Fixity Of Digital Resourcesmentioning

confidence: 99%

Hashes are not suitable to verify fixity of the public archived web

et al. 2023

View full text Add to dashboard Cite

Web archives, such as the Internet Archive, preserve the web and allow access to prior states of web pages. We implicitly trust their versions of archived pages, but as their role moves from preserving curios of the past to facilitating present day adjudication, we are concerned with verifying the fixity of archived web pages, or mementos, to ensure they have always remained unaltered. A widely used technique in digital preservation to verify the fixity of an archived resource is to periodically compute a cryptographic hash value on a resource and then compare it with a previous hash value. If the hash values generated on the same resource are identical, then the fixity of the resource is verified. We tested this process by conducting a study on 16,627 mementos from 17 public web archives. We replayed and downloaded the mementos 39 times using a headless browser over a period of 442 days and generated a hash for each memento after each download, resulting in 39 hashes per memento. The hash is calculated by including not only the content of the base HTML of a memento but also all embedded resources, such as images and style sheets. We expected to always observe the same hash for a memento regardless of the number of downloads. However, our results indicate that 88.45% of mementos produce more than one unique hash value, and about 16% (or one in six) of those mementos always produce different hash values. We identify and quantify the types of changes that cause the same memento to produce different hashes. These results point to the need for defining an archive-aware hashing function, as conventional hashing functions are not suitable for replayed archived web pages.

show abstract

Section: Verifying the Fixity Of Digital Resourcesmentioning

confidence: 99%

Hashes are not suitable to verify fixity of the public archived web

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Alam et al [4] describe archival voids or portions of URI spaces that are not present in a web archive. They created multiple Archival Void profiles using Arquivo.pt access logs, and while doing so, they identified and reported access patterns, status code distributions, and issues such as Soft-404 (when a web server responds with an HTTP 200 OK status code for pages that are actually error pages [37]) through Arquivo.pt server logs.…”

Section: Background and Related Workmentioning

confidence: 99%

Robots Still Outnumber Humans in Web Archives, But Less Than Before

Garg

Alam

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

To identify robots and humans and analyze their respective access patterns, we used the Internet Archive's (IA) Wayback Machine access logs from 2012 and 2019, as well as Arquivo.pt's (Portuguese Web Archive) access logs from 2019. We identified user sessions in the access logs and classified those sessions as human or robot based on their browsing behavior. To better understand how users navigate through the web archives, we evaluated these sessions to discover user access patterns. Based on the two archives and between the two years of IA access logs (2012 vs. 2019), we present a comparison of detected robots vs. humans and their user access patterns and temporal preferences. The total number of robots detected in IA 2012 is greater than in IA 2019 (21% more in requests and 18% more in sessions). Robots account for 98% of requests (97% of sessions) in Arquivo.pt (2019). We found that the robots are almost entirely limited to "Dip" and "Skim" access patterns in IA 2012, but exhibit all the patterns and their combinations in IA 2019. Both humans and robots show a preference for web pages archived in the near past.

show abstract

CDX Summary: Web Archival Collection Insights

Alam

Graham²

2022

Linking Theory and Practice of Digital Libraries

View full text Add to dashboard Cite

Profiling Web Archival Voids for Memento Routing

Cited by 6 publications

References 33 publications

Hashes are not suitable to verify fixity of the public archived web

Hashes are not suitable to verify fixity of the public archived web

Robots Still Outnumber Humans in Web Archives, But Less Than Before

CDX Summary: Web Archival Collection Insights

Contact Info

Product

Resources

About