Hugging Face Xet Integrates Parquet Content-Defined Chunking for Data Efficiency

The Hugging Face Xet storage layer has been integrated with Parquet Content-Defined Chunking (CDC), a feature that significantly improves data efficiency when uploading and downloading Parquet files. This innovation reduces data transfer and storage costs by ensuring only changed data chunks are uploaded or downloaded, rather than entire files.

Apache Parquet is a widely used columnar storage format in data workflows, with Hugging Face hosting over 4 PB of Parquet files. Xet, the platform's content-addressable storage layer, optimizes deduplication by identifying identical data segments. However, traditional Parquet files can produce different byte-level outputs for minor data changes, limiting deduplication effectiveness.

CDC addresses this issue by structuring Parquet files to minimize such differences, aligning with Xet's deduplication capabilities. The feature works by chunking data based on content rather than fixed sizes. When enabled, it ensures that logical column values are consistently divided into data pages, improving the ability of Xet to recognize repeated segments.

This is particularly beneficial for scenarios like adding or removing columns, changing data types, or modifying row groups. For example, re-uploading an exact copy of a Parquet file results in zero data transfer, as Xet identifies identical content. Adding new columns or altering existing ones uploads only the modified sections, while the rest remains deduplicated.

Similarly, changing a column's data type (e.g., from int64 to int32) uploads only the updated column and metadata. Appending new rows to a dataset uploads only the additional data, while inserting or deleting rows—tasks that typically disrupt chunking—now see improved deduplication when CDC is enabled.

Without CDC, such changes can cause entire data pages to shift, leading to higher transfer sizes. With CDC, the impact is minimized, as shown by reduced transfer volumes in tests. Adjusting row-group sizes also benefits from CDC. Smaller or larger row groups, which affect how data is split into pages, still allow efficient deduplication when CDC is used.

This ensures that even with varying configurations, storage costs remain optimized. The feature works across multiple files, enabling efficient deduplication even when datasets are split into different file structures. For instance, uploading a dataset with five, ten, or twenty shards results in minimal additional storage, as Xet identifies shared content.

Pandas users can leverage CDC by setting `use_content_defined_chunking=True` when saving data. This allows for efficient uploads of filtered datasets, such as extracting shorter conversations from a large dataset.

Heatmaps and dedup stats confirmed that CDC reduces transfer size by over 90% in certain cases. The combination of Parquet CDC and Xet is a dream team for data scientists and ML engineers seeking performance at scale.

**Key Facts:**

• Hugging Face introduced Parquet Content-Defined Chunking (CDC) to improve data efficiency when uploading and downloading Parquet files.
• CDC ensures that only modified chunks are uploaded or downloaded, reducing data transfer time and storage costs.
• The feature works by chunking data based on content rather than fixed sizes, improving deduplication effectiveness.
• CDC is particularly beneficial for scenarios like adding or removing columns, changing data types, or modifying row groups.
• Pandas users can leverage CDC by setting `use_content_defined_chunking=True` when saving data.

The integration of Parquet CDC and Xet marks a significant leap in file-level optimization. As the industry continues to grow, this innovation will play a crucial role in reducing storage costs and improving data transfer efficiency.