Intel has released AutoRound, an advanced quantization algorithm for large language models (LLMs) and vision-language models (VLMs), designed to achieve high accuracy at ultra-low bit widths while minimizing tuning. The tool is now integrated into LLM Compressor, a unified system for compression techniques such as quantization and pruning.

What Happened

AutoRound was developed by Intel's team of researchers and engineers, who have been working on improving the accuracy and efficiency of LLMs and VLMs. The algorithm uses signed gradient descent to jointly optimize weight rounding and clipping ranges, enabling accurate low-bit quantization with minimal accuracy loss in most scenarios.

The key strengths of AutoRound include its ability to deliver superior accuracy, especially at very low bit-widths, as well as its support for multiple data types such as W4A16, MXFP8, MXFP4, FP8, and NVFP4. Additionally, the algorithm allows for mixed-bit, layer-wise precision search for flexible accuracy-efficiency trade-offs.

Background and Context

Quantization is a technique used to reduce the size of large language models (LLMs) and vision-language models (VLMs) after training. This process involves representing model weights in lower-precision formats, such as 8-bit integers or floating-point numbers, which can significantly reduce memory usage and improve inference speed.

However, quantization also introduces accuracy loss, especially at very low bit-widths. To address this issue, researchers have been exploring various techniques to optimize the quantization process, including post-training quantization (PTQ) algorithms like AutoRound.

Why It Matters

The integration of AutoRound into LLM Compressor is significant for several reasons. Firstly, it provides a streamlined workflow that allows users to quantize and serve models with just a few lines of code. Secondly, the algorithm's ability to deliver superior accuracy at very low bit-widths makes it an attractive option for applications where memory usage and inference speed are critical.

Furthermore, the integration of AutoRound into LLM Compressor enables seamless compatibility with compressed tensors and direct serving in vLLM. This means that users can take advantage of the algorithm's benefits without having to modify their existing workflows or infrastructure.

What Comes Next

Intel is planning to add native support for FP8, MXFP8, and MXFP4 formats to its next-generation Data Center GPUs, codenamed Crescent Island. This will enable models quantized with AutoRound to naturally scale to take advantage of these data types across the Intel AI hardware portfolio.

The integration of AutoRound into LLM Compressor is just the first step in a broader effort to advance low-bit quantization for LLMs. As researchers continue to explore new techniques and algorithms, it's likely that we'll see even more innovative solutions emerge in the coming months and years.

Key Facts

  • AutoRound is an advanced post-training quantization (PTQ) algorithm developed by Intel for LLMs and VLMs.
  • The algorithm uses signed gradient descent to jointly optimize weight rounding and clipping ranges, enabling accurate low-bit quantization with minimal accuracy loss.
  • AutoRound delivers superior accuracy at very low bit-widths (2-4 bits) while minimizing tuning.
  • The algorithm supports multiple data types, including W4A16, MXFP8, MXFP4, FP8, and NVFP4.
  • AutoRound is now integrated into LLM Compressor, a unified system for compression techniques such as quantization and pruning.