Definition
Quantization maps model weights, activations, or cached values to a smaller set of representable values, usually so each value can be stored with fewer bits. For example, 16-bit floating-point weights can be stored as 8-bit integers with scale information. Other schemes use different low-bit formats.
Quantization can be applied after training or incorporated into training so the model learns to tolerate the lower-precision representation. Different methods quantize different tensors and use different grouping, calibration, and fallback rules.
Simple example
A model whose weights require roughly 28 GB in a 16-bit format may fit into substantially less memory when stored in a suitable 8-bit or 4-bit format. That can make deployment possible on a smaller accelerator or leave more memory for concurrent requests and KV caches.
The team benchmarks the quantized model on its target hardware and evaluates important tasks before replacing the original deployment.
Why it matters
Memory capacity often limits which models can be served and how many requests can run concurrently. Moving less data can also improve throughput or latency when kernels and hardware efficiently support the chosen format.
Some implementations spend time rescaling or converting values during computation. Quantization can also reduce task quality, especially with sensitive layers, outlier values, or aggressive low-bit formats.
One important nuance
Quantization does not automatically make inference faster. A runtime without optimized kernels may spend time repeatedly dequantizing values. The bottleneck may also be work that quantization did not affect. Lower weight memory does not eliminate activation or KV-cache memory either. Measure end-to-end latency, throughput, memory, and task quality on the actual hardware and serving configuration. A smaller model file alone says little about serving speed.