Hugging Face, a leading provider of artificial intelligence and machine learning infrastructure, has published a blog post detailing its efforts to improve the efficiency of Parquet file storage on its Hub platform. The company's developers have been working on optimizing Parquet deduplication, which is critical for storing large datasets without requiring significant additional storage space.

According to Hugging Face, its Hub hosts nearly 11 petabytes (PB) of datasets, with over 2.2 PB of that storage being comprised of Parquet files alone. Optimizing Parquet storage is a high priority for the company, as most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates.

Background and Context

Parquet tables work by splitting the table into row groups, each with a fixed number of rows (for instance 1000 rows). Each column within the row group is then compressed and stored. This means that operations which do not mess with the row grouping, like modifications or appends, should dedupe pretty well.

However, Hugging Face's developers have found that the Parquet layout brings some challenges to deduplication. In a series of experiments, they used a 2GB Parquet file with 1,092,000 rows from the FineWeb dataset and generated visualizations using their dedupe estimator. The results showed that while appending new rows to the file was able to dedupe nearly the entire file, modifications to individual rows resulted in many small regularly spaced sections of new data.

Why it Matters to the Industry

The efficiency of Parquet storage is crucial for large-scale datasets like those hosted on Hugging Face's Hub. With over 11 PB of data stored, optimizing deduplication can lead to significant cost savings and improved performance. In fact, as one commenter pointed out, storing 11 PB on Amazon S3 would cost around $250k per month or $3m per year.

Furthermore, the Hugging Face team's efforts to improve Parquet deduplication have broader implications for the industry. As they noted in their blog post, "we are actively investigating ways to adjust the Parquet format to optimize for dedupe." This could lead to improved storage efficiency and reduced costs for other companies hosting large datasets.

What Comes Next

Hugging Face's developers have proposed a couple of potential solutions to improve Parquet deduplication. One idea is to use relative offsets instead of absolute offsets for file structure data, which would make the Parquet structures position-independent and easy to "memcpy" around. Another suggestion is to support content-defined chunking on row groups, which could be done with minimal blast radius.

The Hugging Face team has also expressed interest in collaborating with the Apache Arrow project to implement some of these ideas in the Parquet/Arrow codebase. This could lead to improved storage efficiency and reduced costs for companies hosting large datasets.

Key Facts

  • Hugging Face hosts nearly 11 PB of datasets, with over 2.2 PB being comprised of Parquet files alone.
  • Optimizing Parquet storage is a high priority for Hugging Face due to the large size of its datasets.
  • The company's developers have been working on improving Parquet deduplication through experiments and proposed solutions.
  • Hugging Face has expressed interest in collaborating with the Apache Arrow project to implement improvements to Parquet/Arrow codebase.
  • Storing 11 PB on Amazon S3 would cost around $250k per month or $3m per year.

In conclusion, Hugging Face's efforts to improve Parquet deduplication have significant implications for the industry. By optimizing storage efficiency and reducing costs, companies hosting large datasets can benefit from improved performance and reduced expenses. As the company continues to work on improving its infrastructure, it will be interesting to see how these developments impact the wider industry.