Quantization is the dominant practical technique for improving LLM inference efficiency — more impactful than batching, more accessible than custom kernels, and applicable to virtually every production deployment. Despite this, the production understanding of quantization in most engineering teams is shallow: "lower precision, lower cost, some quality loss, use INT8." The actual tradeoff space is significantly more nuanced, and the difference between a well-calibrated quantization strategy and a poorly chosen one can be 40–60% cost on equivalent hardware.
This is a technical view of where the quantization efficiency frontier sits today — what methods are available, what their actual cost-quality tradeoffs look like in production workloads, and what the active areas of development are that will change the calculus over the next 12–18 months.
Why quantization works for transformers specifically
Quantization — representing neural network weights and activations in lower-precision numerical formats — has been studied for neural networks generally since the mid-2010s. For transformers specifically, quantization works surprisingly well because of a structural property of how transformer layers process information: the information capacity of each layer is distributed broadly across many weight values, rather than concentrated in a small number of high-precision values. This means that reducing the precision of individual weights by a moderate amount (FP16 to INT8) has a small effect on the aggregated information capacity of the layer, not a proportional one.
The exception to this is outlier activations — values in intermediate computations that are significantly larger than the typical range. LLM activations have a fat-tailed distribution, and the outliers are disproportionately important for model quality. Standard quantization schemes that use a uniform quantization grid to represent all activation values suffer disproportionate quality degradation because the high-magnitude outliers force a wide quantization range that leaves low-magnitude values with poor resolution. Most of the research progress in LLM quantization over the past two years has been about handling these outliers better.
The method landscape: weight-only vs. activation quantization
Weight-only quantization (GPTQ, AWQ, GGUF) quantizes model weights but keeps activations in high precision. The advantage is simplicity and robust quality: since activations stay in FP16, the outlier problem does not apply. The disadvantage is that the compute operations (matrix multiplications) are still performed in FP16 — the GPU does not gain the full benefit of lower-precision arithmetic, only the memory bandwidth benefit from storing smaller weight values. On memory-bandwidth-bound workloads (large batches, long sequences), this is still a significant gain: 4-bit weight quantization reduces weight memory by 4x and increases the number of tokens that can be generated per second by 1.5x to 2.5x, depending on hardware. GPTQ and AWQ are the most mature weight-only methods and the ones I would recommend for production deployments today.
Activation quantization (SmoothQuant, FP8) quantizes both weights and activations, allowing the GPU to use lower-precision compute units that operate faster. NVIDIA's H100 has hardware FP8 tensor cores that are approximately 2x faster than FP16 tensor cores for matrix multiply operations. FP8 activation quantization, when it can be applied without quality regression, can therefore deliver 2x throughput improvement on compute-bound workloads. The challenge is managing the outlier activations — SmoothQuant addresses this by mathematically "smoothing" the outlier distribution across weights and activations before quantization, making the resulting activation distributions more uniform and quantization-friendly.
KV cache quantization is an underexplored dimension. The KV cache — the stored attention key-value matrices for processed context — is a major consumer of GPU memory in long-context inference, and it grows linearly with sequence length and batch size. Quantizing the KV cache from FP16 to INT8 or INT4 can reduce memory pressure significantly without the quality risk of weight quantization, because the KV cache values are intermediate computations that do not accumulate error in the same way as low-precision weight storage. This is an active area with significant practical upside.
What the quality tradeoffs actually look like in production
The academic benchmark results for quantization (perplexity on common corpora, accuracy on standard NLP benchmarks) are broadly trustworthy but not always predictive of production quality for specific workloads. The quality tradeoff depends heavily on the task type.
For generation tasks (summarization, creative writing, general Q&A), INT8 weight quantization is essentially quality-neutral for models above 13B parameters — the quality regression is below human-perceptible thresholds in controlled evaluations. INT4 weight quantization shows measurable regression on reasoning-intensive tasks but is acceptable for generation quality. For models below 7B parameters, quantization quality regression is more significant — the base model has less redundancy to absorb the precision reduction.
For precise extraction tasks (structured data extraction, code generation, mathematical reasoning), quantization quality regression is more noticeable. A team extracting structured information from financial documents should evaluate quantized model quality carefully against their specific extraction schema before deploying INT4 weight quantization in production. The regression is typically 5–10% on precision metrics for INT4, which is often acceptable but occasionally is not.
The active frontier: per-layer and mixed-precision schemes
The next generation of quantization methods is moving beyond uniform precision assignments (all weights at INT8, or all at INT4) toward mixed-precision schemes that assign different precision levels to different layers or weight matrices based on their sensitivity to quantization error. Early transformers layers tend to be more sensitive; later layers tend to be more robust. Attention projection weights and MLP expansion weights have different outlier profiles. Per-layer sensitivity analysis can guide precision assignment in ways that maintain near-FP16 quality while achieving sub-INT4 average precision.
This is currently an area of active research — SpQR, QuIP, and related methods are producing results that push the quality-efficiency frontier meaningfully. The production tooling to deploy mixed-precision models efficiently is still maturing, but the direction is clear: the binary choice between FP16 quality and INT4 cost will be replaced by a continuum of quality-cost configurations selected per-model and per-task. The teams and companies building tooling for this continuum are working on the right problem at the right time.