Key Insights

  • Optimized for LLM decoding
  • Stored in 3bit packed format, do LUT dequantize on the fly, calculated in MXFP8
  • Weight only 3-bit, 3.375 bits-per-weight, 20% gains from mxfp4
  • N=8, K=64 forms a block, sharing one codebook (8 entries of FP8 codeword)
  • The micro-scaling of MXFP8 is also valid as K=32 per-group scaling

Current Status:

  • Hardware: Supported on Rubin
  • Software: Developer preview in CUDA13.41

Comparison with other numeric format

Bits per weights:

Format Element / index bits Scaling bits Codebook bits Total bits Short explanation
MXFP8 8 8/32 = 0.25 0 8.25 FP8 values with one E8M0 scale per 32 elements.
NVFP4 4 8/16 = 0.5 0 4.5 E2M1 FP4 values with one E4M3 scale per 16 elements; a tensor-level FP32 scale is additional.
MXFP4 4 8/32 = 0.25 0 4.25 E2M1 FP4 values with one E8M0 scale per 32 elements.
3bitLUT-MXFP8 3 8/32 = 0.25 (8 × 8)/512 = 0.125 3.375 Packed indices select one of eight FP8 codewords shared by 512 weights, with MXFP8 block scaling.

All numeric columns are amortized bits per weight. The totals follow the accounting above and exclude padding and tensor-level metadata.21

Benchmark Results

No figures for now

Tensor Core MMA Overview

The compressed 3bit weight is feed into Tensor Core directly, so memory bandwidth and capacity can be saved on both HBM and SRAM level.

However the peak computation should not increase compared with MXFP8, if the workload already hits compute bound roofline (which is unusual)

NVIDIA MMA layout showing compressed B indices in shared memory and LUT metadata in Tensor Memory.
NVIDIA PTX: matrix B usage in computation. Compressed indices and LUT metadata are separate inputs. Scroll horizontally on narrow screens.

Quantize granularity

N=8,K=64(in total 512 elements) shares a single codebook(LUT)1

when combined with MXFP8 format, the K=64 can be splitted into 2 scaling groups, each scaled with a FP8-E8M0 scaling factor.

NVIDIA diagram showing an N=8, K=64 tile represented by three-bit indices and an eight-entry LUT.
NVIDIA PTX: one eight-entry LUT for an N=8, K=64 tile. Scroll horizontally on narrow screens.

Codebook storage

CodeBook is stored in 128Byte-aligned format (cache friendly), thus 16 LUT-blocks should maximize the throughput. The 16-blocks is organized as N=64, K=128

NVIDIA GMEM and TMEM codebook layouts with sixteen eight-byte LUTs in each colored group.
NVIDIA PTX: GMEM and TMEM layout of the lookup table. Each colored group contains sixteen LUTs. Scroll horizontally on narrow screens.

References

  1. NVIDIA, PTX ISA 9.4 — Decompression of input matrices, CUDA 13.4 developer preview. Source of the three diagrams above. ↩ ↩2 ↩3

  2. NVIDIA, Introducing NVFP4 for Efficient and Accurate Low-Precision Inference: source for the format descriptions in the comparison table. ↩