The Case For Alternative Web Archival Formats To Expedite The Data-To-Insight Cycle

Wang, Xinyue; Xie, Zhiwu

doi:10.1145/3383583.3398542

Cited by 7 publications

(5 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7) Experiment with column-oriented formats for archive indexes -In addition to formats that are common within the web archiving field, like WARC, WAT, WET, and CDX, other formats such as Parquet, which is used in cloud-based or big-data oriented environments, are making inroads into the field of web archiving [16]. Apache Parquet files are a column-oriented file type that is used to store and retrieve data more efficiently.…”

Section: Methodsmentioning

confidence: 99%

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

Phillips,

Alam

2023

2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

View full text Add to dashboard Cite

The End of Term (EOT) Web Archive Dataset presents a longitudinal dataset of the US federal web which includes publicly available .gov and .mil domains, created during the 2008, 2012, 2016, and 2020 presidential elections in the United States. Based on the End of Term Web Archive, this dataset presents 461TB of WARC data and accompanying derivative files in WAT, WET, and CDX format. A metadata sidecar file is also provided that contains content-based characterizations, including languages, character sets, format identifiers, and mime types. In addition to these derivative formats, CDX indexes in the ZipNum and Parquet formats that provide additional functionality to the dataset are included. The EOT dataset is freely available on the Amazon S3 platform as part of the Amazon Open Data Program.

show abstract

Section: Methodsmentioning

confidence: 99%

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

Phillips,

Alam

2023

2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

View full text Add to dashboard Cite

show abstract

“…Interoperability between datasets, formats, and software tools is essential for integrating datasets from multiple sources, and is a continuing challenge for knowledge infrastructures. The FITS format and tools layered upon these standards are showing their age, raising concerns among astronomers for future interoperability (Mink, 2015;Thomas et al, 2015;Wang & Xie, 2020). New tools such as Jupyter Notebooks provide scientists the ability to release executable packages of data, pipelines, and workflows.…”

Section: Theme 4: Using and Reusing Data Productsmentioning

confidence: 99%

From Data Processes to Data Products: Knowledge Infrastructures in Astronomy

Borgman¹,

Wofford²

2021

Harvard Data Science Review

View full text Add to dashboard Cite

We explore how astronomers take observational data from telescopes, process them into usable scientific data products, curate them for later use, and reuse data for further inquiry. Astronomers have invested heavily in knowledge infrastructuresrobust networks of people, artifacts, and institutions that generate, share, and maintain specific knowledge about the human and natural worlds. Drawing upon a decade of interviews and ethnography, this article compares how three astronomy groups capture, process, and archive data, and for whom. The Sloan Digital Sky Survey is a mission with a dedicated telescope and instruments, while the Black Hole Group and Integrative Astronomy Group (both pseudonyms) are university-based, investigator-led collaborations. Findings are organized into four themes: (1) how these projects develop and maintain their workflows; (2) how they capture and archive their data; (3) how they maintain and repair knowledge infrastructures; and (4) how they use and reuse data products over time. We found that astronomers encode their

show abstract

“…The study proceeds from an experimental assessment of two formats in the absence of a specific task of choosing alternatives. The authors of [27] pursue the goal of finding an alternative for the WARC format when developing web services. Apache Parquet and Apache Avro are also alternatives in this study.…”

Section: E Analysis Of Data Storage Formatsmentioning

confidence: 99%

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

Belov¹,

Nikulchev²

2021

IJACSA

View full text Add to dashboard Cite

When developing large data processing systems, the question of data storage arises. One of the modern tools for solving this problem is the so-called data lakes. Many implementations of data lakes use Apache Hadoop as a basic platform. Hadoop does not have a default data storage format, which leads to the task of choosing a data format when designing a data processing system. To solve this problem, it is necessary to proceed from the results of the assessment according to several criteria. In turn, experimental evaluation does not always give a complete understanding of the possibilities for working with a particular data storage format. In this case, it is necessary to study the features of the format, its internal structure, recommendations for use, etc. The article describes the features of both widely used data storage formats and the currently gaining popularity.

show abstract

The Case For Alternative Web Archival Formats To Expedite The Data-To-Insight Cycle

Cited by 7 publications

References 37 publications

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

End of Term Web Archive Dataset: Longitudinal Web Archive of .GOV and .MIL Domains

From Data Processes to Data Products: Knowledge Infrastructures in Astronomy

Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform

Contact Info

Product

Resources

About