The Pile

The Pile is an 825 GiB English-language text corpus assembled by EleutherAI for training large language models, released on December 31, 2020 alongside the paper "The Pile: An 800GB Dataset of Diverse Text for Language Modeling."¹ Built from 22 curated sub-datasets, it was intended to give the collective's open GPT-3 replication effort a documented, publicly available alternative to the undisclosed corpora behind proprietary models, on the argument that greater dataset diversity improves cross-domain knowledge and downstream generalization.¹ The Pile became the training corpus for EleutherAI's model line, including GPT-Neo, GPT-J, GPT-NeoX-20B, and Pythia, and was adopted in whole or in part by several outside labs. From 2023 its distribution was curtailed by copyright takedowns aimed at the Books3 component, and in 2025 EleutherAI released the Common Pile, built entirely from openly licensed and public-domain text, as its successor.

Composition

The Pile's largest component, Pile-CC, is a filtered subset of Common Crawl web scrapes; the remaining 21 components were chosen for quality and topical range.¹ Academic and scientific text comes from PubMed Central, arXiv, PubMed abstracts, PhilPapers, and NIH ExPORTER grant abstracts. Books are represented by Books3, BookCorpus2, and Project Gutenberg. Technical, legal, and governmental material includes GitHub code, FreeLaw court opinions, USPTO patent background sections, and EuroParl parliamentary proceedings, while conversational and internet text is drawn from Stack Exchange, Hacker News, Ubuntu IRC logs, the Enron email corpus, OpenSubtitles, and YouTube subtitles. The set is rounded out by English Wikipedia, OpenWebText2 (web pages linked from Reddit submissions), and DM Mathematics, a synthetically generated mathematics dataset originating with DeepMind.

Rather than sampling components in proportion to raw size, the authors assigned each an epoch count, so that sources they judged higher quality recur during a full pass over the corpus; the 825 GiB figure (about 886 GB in decimal units, rounded to "800GB" in the paper's title) refers to the underlying data before this up-weighting.¹ A companion datasheet published in January 2022 documents the provenance, collection, and processing decisions behind each component.²

Design and evaluation

Per its authors, the Pile was a response to the practice of training ever-larger models on minimally processed web scrapes: a deliberately diverse, curated corpus was expected to yield models with broader cross-domain knowledge and better downstream generalization.¹ In the paper's evaluations, GPT-2 and GPT-3 performed comparatively poorly on several components, notably academic writing, which the authors took as evidence that the Pile covers text under-represented in earlier training distributions; models trained on the Pile showed significant improvements over baselines trained on raw Common Crawl and on CC-100.¹

Adoption

Within EleutherAI, the Pile served as the training set for the collective's full model progression, from GPT-Neo and GPT-J through GPT-NeoX-20B, whose tokenizer was also trained on the corpus.³ The 2023 Pythia suite trained eight model sizes, from 70 million to 12 billion parameters, on the standard Pile and the same eight sizes again on a deduplicated variant, with checkpoints and exact data ordering released to support research on memorization and training dynamics.⁴

Outside the collective, Cerebras trained its 2023 Cerebras-GPT family of open models on the Pile following Chinchilla compute-optimal scaling rules,⁵ and the 530-billion-parameter Megatron-Turing NLG model from Microsoft and NVIDIA built the bulk of its training corpus from the Pile components its authors judged highest quality, including Pile-CC, Books3, and OpenWebText2.⁶ Yandex's 100-billion-parameter YaLM-100B (June 2022) drew 25 percent of its 1.7 TB training corpus from the Pile, per its repository documentation.¹⁴ Distribution ran through direct downloads hosted by the archive site The Eye and an official repository on Hugging Face, which lists hundreds of hosted models as trained on the dataset.

Books3 and copyright disputes

Books3, one of the corpus's largest components, was a roughly 37 GB plain-text collection of about 196,640 books compiled by the developer Shawn Presser from the contents of Bibliotik, a private shadow-library tracker, and first published through The Eye in late 2020.⁷ The Pile paper was open about this origin,¹ and Presser framed the release as a way to let open developers match labs training on undisclosed book corpora, such as the "Books1" and "Books2" datasets behind OpenAI's GPT-3.¹⁰ The component made the Pile a recurring reference point in disputes over copyright and training data; reporting on the dataset noted its use in training Meta's models,⁷ and the paper for the original LLaMA lists the Books3 section of the Pile among its pre-training sources.¹¹

In mid-2023 the Danish anti-piracy group Rights Alliance (Rettighedsalliancen), acting for rights holders whose titles appeared in the collection, issued DMCA takedown notices;⁷ the group said the collection included roughly 150 titles from its member publishers,¹² and The Eye removed Books3 in mid-July.¹³

The takedowns effectively ended official distribution of the full corpus. The Pile's direct downloads were hosted on The Eye, and the official Hugging Face repository fetches its files from that host through a loading script, so with the files removed the canonical download path no longer functions. A community mirror, ''Pile-uncopyrighted'', which strips out Books3, BookCorpus2, OpenSubtitles, YouTube subtitles, and OpenWebText2 (the components not explicitly permitted for use in AI training, per its maintainer), became the de facto access path for the remaining roughly 335 GB.⁸

Successor

In June 2025, EleutherAI and academic collaborators released the Common Pile v0.1, an 8 TB corpus assembled from 30 sources of public-domain and openly licensed text, positioned as a first step toward addressing the licensing problems that had overtaken its predecessor.⁹ Two 7-billion-parameter models trained on it, the Comma v0.1 series, were reported by the authors to be competitive with models trained on unlicensed text at similar compute budgets, such as LLaMA 7B and Llama 2 7B.⁹

References