The Pile: the open-source 825 GiB dataset for training LLMs
In one sentence EleutherAI publishes The Pile, an 825 GiB English text corpus made of 22 sub-datasets (filtered Common Crawl, PubMed, Books3, arXiv, GitHub, Stack Exchange and more), designed for pre-training open language models.
Training a GPT-3-class model needs a huge amount of text. OpenAI never published its dataset, which makes it hard for others to reproduce its results.
EleutherAI, a collective of volunteer researchers, publishes its answer: The Pile, 825 GiB of downloadable text. It isn't just web pages: it mixes scientific papers, open-source code, books, Stack Exchange threads, subtitles, patents, court rulings, emails.
It becomes the base of EleutherAI's first open models (GPT-Neo, GPT-J, GPT-NeoX) and of much language-model research.
Companies
EleutherAI
Tools
The Pile
Tags
Sources