Hugging Face's Xet team has developed a new storage approach called content-defined chunking (CDC), which breaks down large files into variable-sized chunks, improving storage efficiency and iteration speed. This method is being tested on the Hugging Face Hub, where it has shown significant reductions in storage requirements and upload/download times.

Background and Context

The Hugging Face Hub stores over 30 petabytes of models, datasets, and spaces in Git LFS repositories. However, this approach has limitations when dealing with large files, as any change to a file requires re-uploading the full asset. This can be expensive, especially considering that average Parquet and CSV files on the Hub range between 200-300 MB, while Safetensor files are around 1 GB, and GGUF files can exceed 8 GB.

To illustrate this issue, consider a file with a single line of metadata modified. In a traditional Git LFS setup, the entire file would need to be re-uploaded, resulting in significant user time and transfer costs. Moreover, Git LFS also needs to save full versions of both files, leading to bloated storage costs.

How CDC Works

CDC breaks down large files into variable-sized chunks using a rolling hash algorithm that scans the file's byte sequence. This approach is more efficient than traditional methods because it only transfers modified chunks, reducing storage requirements and upload/download times. The content of these chunks is hashed to create a mapping between chunk hash and bytes, which will eventually be stored in a content-addressed store (CAS).

For example, consider a file with the contents "transformerstransformerstransformers". A rolling hash algorithm would compute a hash over a sliding window of data, breaking down the file into chunks based on predefined conditions. In this case, the condition is that the hash satisfies a predefined condition, such as hash(data) % 2^12 == 0. If the sequence "mers" produces a hash that meets this condition, the file will be split into three identical chunks.

Benefits and Impact on the Industry

The benefits of CDC are significant, particularly for large files stored on the Hugging Face Hub. By only transferring modified chunks, users can reduce storage requirements and upload/download times. This approach also provides tools for deeper optimizations and benefits, changing how models and datasets are managed for the entire Hugging Face community.

According to a benchmark conducted by the Xet team, CDC showed a consistent 50% improvement in storage and transfer performance across three iterative development use cases. One example was the CORD-19 dataset, a collection of COVID-19 research papers curated between 2020 and 2022 with 50 incremental updates.

What Comes Next

The Xet team is currently working on integrating CDC into the Hugging Face Hub, aiming to roll out some Xet-backed repositories in early 2025. This development has significant implications for the adult industry, where large files are common and storage efficiency is crucial.

Key Facts

  • Hugging Face stores over 30 petabytes of models, datasets, and spaces in Git LFS repositories.
  • CDC breaks down large files into variable-sized chunks using a rolling hash algorithm.
  • The Xet team has shown a consistent 50% improvement in storage and transfer performance with CDC.
  • CDC reduces storage requirements by only transferring modified chunks.
  • Parquet files can be effectively deduplicated using CDC, reducing storage requirements by up to 75.6 GB.

The integration of CDC into the Hugging Face Hub has the potential to revolutionize how large files are stored and managed in the adult industry. By reducing storage requirements and upload/download times, this approach can help operators save costs and improve efficiency.