Hugging Face Unveils Cosmopedia: Largest Open Synthetic Dataset for Language Model Pre-Training

The Hugging Face team has developed Cosmopedia, a large-scale synthetic dataset designed for pre-training language models, aiming to replicate the success of Microsoft's Phi models. The dataset contains over 30 million files and 25 billion tokens, making it the largest open synthetic dataset to date.

Background and Context

The development of Cosmopedia marks an important step towards advancing large language models. Traditionally, creating datasets for supervised fine-tuning and instruction-tuning required hiring human annotators, a time-consuming and expensive process that limited the development of such datasets to a few key players in the field.

However, with the recent advancements in synthetic data generation, hundreds of high-quality synthetic fine-tuning datasets have been developed using models like GPT-3.5 and GPT-4. The community has also supported this development with numerous publications that guide the process for various domains and address associated challenges.

The Phi models developed by Microsoft were pioneers in this area, relying heavily on synthetic data for training and surpassing larger models trained on web datasets for longer periods. However, the technical reports of the Phi models leave out substantial details regarding the curation of their synthetic training datasets, sparking debate among enthusiasts and skeptics alike.

Why Cosmopedia Matters to the Industry

Cosmopedia's ultimate goal is to provide an enormous amount of comprehensive synthetic data of excellent quality. This dataset can be used for pre-training language models from scratch, presenting a unique set of challenges that require specific methods to select a wide array of prompts related to different topics.

The team employed two approaches to build Cosmopedia's prompts: conditioning on curated sources and conditioning on web data. They called this "seed data," the original set of information used to create their conditions. This approach ensures diversity in the subjects covered, which is critical for maximum performance.

Curated Sources: The team used topics from reputable educational resources, including OpenStax, WikiHow, Stanford courses, and Khan Academy. However, this approach has a limitation - it cannot scale, even though it produces high-quality content.

The Technical Stack Behind Cosmopedia

The development of Cosmopedia involved several technical tools, including text clustering, text generation at scale, and training the Cosmo-1B model. The team used the llm-swarm library to generate 25 billion tokens of synthetic content using Mixtral-8x7B-Instruct-v0.1.

They also employed a decontamination pipeline to ensure their dataset is free of any samples from the test benchmarks. This process involves identifying potentially contaminated samples using a 10-gram overlap and comparing the dataset sample against the benchmark sample.

Key Facts

• Cosmopedia contains over 30 million files and 25 billion tokens, making it the largest open synthetic dataset to date.
• The team used two approaches to build Cosmopedia's prompts: conditioning on curated sources and conditioning on web data.
• Curated Sources include topics from reputable educational resources like OpenStax, WikiHow, Stanford courses, and Khan Academy.
• Web Data accounts for more than 80% of Cosmopedia's prompts, making it the most scalable approach.
• The team employed a decontamination pipeline to ensure their dataset is free of any samples from the test benchmarks.

What Comes Next?

The development of Cosmopedia marks an important step towards advancing large language models. However, there is potential for further improvement of the dataset, which can be achieved by refining the data selection processes and improving the training protocols.

The team is actively working on enhancing the quality of the generated content, including strategies to mitigate hallucinations and improve the accuracy and reliability of the generations.