Datasheet for the Pile

Abstract

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online. Background on the PileThe Pile is a massive text corpus created by EleutherAI for large-scale language modeling efforts. It is comprised of textual data from 22 sources (see below) and can be downloaded from the official website as well as from a community mirror. Each source dataset is at its core a textual work, and any non-textual data (including metadata) has been removed. While still preserving their internal order, the documents from all the sources have been randomly shuffled. For further information on the Pile, see Gao et al. [2020].This document is not intended to be -and should not be used as -a substitute for a datasheet for the original versions of the component datasets. While it is accurate for the text data that we derived from each component dataset, the original source dataset may have other properties. This document is intended to inform people interested in using the Pile for natural language processing. People interested in using the original datasets should contact the data owners for information about the properties of the original data.It is not always the case that the answer to the questions below are known with certainty. For example, while we have no reason to believe that personal identifying information (PII) is contained in most of the subsets of our dataset, it is always possible that someone wrote down PII in a document and uploaded it to arXiv. Due to the sheer scale of the data, it is impractical to systematically search through every text to validate that it is what it purports to be. We have endeavored to answer the questions below as best we can, and to be open and honest about the limitations of the accuracy of this document. Anyone who engages in research on or with the Pile is welcome to contact us to have their findings added to this document. Similarly, we welcome all comments, suggestions, or corrections. Datasets contained in the Pile:Pile-CC: The Pile-CC dataset is a sample from the Common Crawl WARCs that has been converted to text using jusText [Endrédy and Novák, 2013].

Data Governance in the Age of Large-Scale Data-Driven Language Technology

“…For LLMs in particular, the data used to train them are one further step removed from the task-specific models built from them, so the link between data and ML progress is even more abstracted [87,116]. Second, research addressing dataset choices, creation, and curation, is systematically "under-valued and de-glamorised" [3,123] 16 . Even works that do include significant curation efforts for the sake of improving models [57,113] focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes [19].…”

“…One approach put forward in recent years to foster more accountability of these data practices has been documentation standards for data and models in natural language processing [11,59] and ML in general [93]. There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102,107,121,123], with several noteworthy initiatives aiming to document both existing [9,20,30,43], and newly developed [16,60,136] resources.…”

“…15 The current proposal for the EU AI Act [50] distinguishes between application areas on the basis of risk they pose, and would institute external "conformity assessments" for the more risky applications. 16 For a direct example of how the ML community treats work on datasets and values, see reviews for [3] here 17 mostly geared towards code and experiment tracking, but also covering training and evaluation data 10 Data Governance in the Age of Large-Scale Data-Driven Language Technology FAcct '22, 2022, Seoul, South Korea little attention is paid to the later stages of the data life cycle (see Section 2.3), or to data management models that intentionally include data subjects. We review common approaches to hosting and distributing ML data in Section 4.…”

“…SURVIVAL The provisions set forth in section 6(b) (Limits of Liability), 10 (Term and Termination), 12 (Confidentiality), 15 (Governing Law), 16 (Survival) and Exhibit A (Section Restrictions of Use in the Dataset section) shall survive the termination of this agreement and continue to bind both parties. 16. SEVERABILITY If any provision of this Agreement is held to be invalid, illegal or unenforceable, the remaining provisions shall be unaffected thereby and remain valid as if such provision had not been set forth herein.…”

Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources

“…Common Crawl, meanwhile, has been shown to contain hate speech and over-represent sexually explicit content (Luccioni and Viviano, 2021), and typical web-crawling collection practices have no structures for supporting informed consent beyond websites' own terms and conditions policies that users rarely read (Cakebread, 2017;Obar and Oeldorf-Hirsch, 2020). Several documentation schemas for natural language processing (NLP) datasets (Bender and Friedman, 2018;Gebru et al, 2018;Gebru et al, 2021;Holland et al, 2018;Pushkarna et al, 2021) have been recently produced to aid NLP researchers in documenting their own datasets (Gao et al, 2020;Biderman et al, 1 http://commoncrawl.org/ 2022; Gehrmann et al, 2021;Wang et al, 2021) and even to retrospectively document and analyze datasets that were developed and released by others without thorough documentation (Bandy and Vincent, 2021;Kreutzer et al, 2021;Birhane et al, 2021;Dodge et al, 2021). Data documentation to support transparency has gained traction following calls for a reevaluation of the treatment of data in machine learning (ML) at large (Prabhu and Birhane, 2020;Jo and Gebru, 2020;Paullada et al, 2021;Gebru et al, 2021).…”