CDX Summary: Web Archival Collection Insights

Alam, Sawood; Graham, Mark

doi:10.1007/978-3-031-16802-4_25

Cited by 1 publication

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work is expected to continue to leverage existing tools and processes developed by Common Crawl for graph generation for this portion of the work. With the complete dataset available in CDX format, overviews of each of the EOT crawl years using CDX summarization tools [1] can be generated. These can be helpful in communicating the contents of this dataset to others.…”

Section: Crawlmentioning

confidence: 99%

See 1 more Smart Citation

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

Phillips,

Alam

2023

2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

Self Cite

View full text Add to dashboard Cite

The End of Term (EOT) Web Archive Dataset presents a longitudinal dataset of the US federal web which includes publicly available .gov and .mil domains, created during the 2008, 2012, 2016, and 2020 presidential elections in the United States. Based on the End of Term Web Archive, this dataset presents 461TB of WARC data and accompanying derivative files in WAT, WET, and CDX format. A metadata sidecar file is also provided that contains content-based characterizations, including languages, character sets, format identifiers, and mime types. In addition to these derivative formats, CDX indexes in the ZipNum and Parquet formats that provide additional functionality to the dataset are included. The EOT dataset is freely available on the Amazon S3 platform as part of the Amazon Open Data Program.

show abstract

Section: Crawlmentioning

confidence: 99%

“…The full dataset is available with a Creative Commons CC0 1.0 Universal (CC0 1.0) 1 Public Domain Dedication and is downloadable from the End of Term Website in the data section 2 . A record for the dataset is also available in the Registry of Open Data on AWS 3 .…”

Section: Introductionmentioning

confidence: 99%