Understanding IQ (Importance Matrix Quants) and NVFP4

In the rapidly evolving field of artificial intelligence, running large language models (LLMs) efficiently is a major challenge. Two quantization techniques have emerged as key solutions: Importance Matrix Quants (IQ) and NVFP4. While both aim to reduce model size and accelerate inference, they operate on fundamentally different principles and target different hardware. This article explains what they are, how they perform, and why your choice depends heavily on your GPU.

What are Importance Matrix Quants (IQ)?

Importance Matrix Quants (IQ) are a software-based quantization method primarily used within the llama.cpp ecosystem. The core idea is to use an 'importance matrix' (imatrix) generated by running a calibration dataset through the model. This matrix identifies which weights are critical for the model's intelligence and which are less significant. During quantization, IQ protects the important weights by keeping them at higher precision while aggressively compressing the less important ones. This allows the model to maintain higher coherence and accuracy at very low bit-rates, such as 2-bit or 3-bit, compared to standard quantization methods.

IQ is a software-level optimization. It does not require specific hardware features to function; it works on CPUs and almost any GPU. It is the preferred choice for users who need to run large models on limited hardware, for example, running a 70B model on a home PC with limited VRAM. The trade-off is that IQ quantization can be computationally intensive to create, and the resulting model may still have some quality loss, especially at extremely low bitrates. However, for many use cases, it offers an excellent balance between size and performance.

What is NVFP4?

NVFP4 is a hardware-accelerated 4-bit floating-point format introduced specifically for the NVIDIA Blackwell architecture, such as the B200 and RTX 50-series GPUs. It is designed for maximum throughput and efficiency in data centers and high-end workstations. NVFP4 uses a proprietary 4-bit floating-point format (E2M1 encoding) with a two-level scaling strategy: a fine-grained FP8 scale factor for every 16-element block, and a per-tensor FP32 scale factor.

Because Blackwell Tensor Cores have native hardware support for this format, they can perform matrix multiplication directly on these 4-bit values. This eliminates the need to dequantize weights to a higher precision (like FP16 or BF16) during computation. As a result, NVFP4 provides massive performance gains—often 2–3x faster than FP8 and significantly more memory-efficient than FP16—because it reduces data movement and allows the GPU to process more operations per clock cycle. The precision of NVFP4 is 4-bit floating point, which is lower than FP16 but sufficient for many inference tasks, especially when combined with advanced scaling techniques.

Hardware Support and Emulation on Older GPUs

The fundamental difference between these techniques lies in hardware support. On Blackwell GPUs (e.g., RTX 50-series, B200), NVFP4 runs natively. The GPU processes the 4-bit data directly, delivering full speed and memory benefits. For instance, a 70B model quantized with NVFP4 can fit into significantly less VRAM and run much faster than its FP16 counterpart.

However, on older architectures like the NVIDIA RTX 3000 (Ampere) and 4000 (Ada Lovelace) series, these cards lack native FP4 Tensor Cores. While you can technically load an NVFP4-quantized model file on these cards, the GPU cannot perform the math in 4-bit. Instead, the software must 'emulate' the format by dequantizing the weights back to a higher precision (like BF16 or FP16) before the calculation can occur. The result: you lose all the performance and memory benefits. The model will consume the same amount of VRAM as a standard FP16 model, and the inference speed will be no faster than a standard model. There is no emulation that grants the speed or memory efficiency of NVFP4 on older hardware; it is effectively just a storage format that must be unpacked to be used.

Comparison: IQ vs. NVFP4

The following table summarizes the key differences between IQ and NVFP4.

Conclusion

In summary, IQ and NVFP4 serve different purposes. IQ is a versatile software solution that excels at compressing models for memory-constrained environments, regardless of hardware. NVFP4 is a hardware-specific format that unlocks unprecedented speed and efficiency on NVIDIA Blackwell GPUs. If you own a Blackwell card, NVFP4 can dramatically improve your inference performance. If you are on older hardware, IQ remains a powerful tool for running large models within your VRAM limits. Understanding these differences will help you choose the right quantization strategy for your AI workloads.