LLM Quantization for GPU Deployment 2026 — GPTQ vs AWQ vs FP8 vs NVFP4

When a B2B buyer specs a GPU server for LLM inference, the first question is almost always "how many GPUs, and which one?" The answer changes dramatically depending on quantization. A 70B model that appears to require two H100s at FP16 can run on a single 48 GB card at 4-bit with a 1–3% quality difference — that is a roughly $20,000 purchasing swing driven entirely by software choices. This guide covers the four formats that matter in 2026 — GPTQ, AWQ, FP8 and NVFP4 — and maps them onto the hardware you actually buy.

Why Quantization Is a Hardware Purchase Decision

LLM inference is bound by VRAM capacity and memory bandwidth, not raw FLOPs. Quantization attacks both: it shrinks weights so more model fits per card, and it cuts bytes-per-token so each card serves tokens faster. For a buyer, the practical result is that the quantized model size — not the published FP16 size — decides the GPU count and generation. Get this wrong and you either over-buy (GPUs that sit half-empty) or under-buy (a model that does not fit in memory at all). Get it right and a quantized deployment routinely cuts hardware cost 2–4× at near-identical output quality.

The Four Formats That Matter in 2026

GPTQ and AWQ are post-training weight-only methods: you run them once on a copy of the model with a small calibration set, and the result is a smaller file you deploy. AWQ generally holds quality slightly better than GPTQ at 4-bit on larger models, at the cost of slightly more complex kernels. FP8 is the lossless production default on Hopper and Ada — no calibration, no quality debate, native tensor-core acceleration. NVFP4 is Blackwell's answer: 4-bit floating point with a per-block scale, roughly 2× the throughput of FP8 on B200-class parts.

VRAM Math — What Actually Fits on Which GPU

Weights-only figures below (add 10–30% for KV cache and activations depending on context length). A 128K-token KV cache on a 70B model can add 10–16 GB at FP16, which is why KV-cache quantization (FP8 or INT8 keys/values) matters in production — most serving stacks now support it.

Quality and Speed Trade-Offs in Production

• INT8 / FP8 — essentially lossless for inference, typically <0.5% on standard benchmarks. If your serving stack supports FP8 (TensorRT-LLM, vLLM), it is the safe default on Hopper/Ada hardware.
• 4-bit (GPTQ/AWQ) — 1–3% quality loss on most tasks; AWQ is the stronger choice for instruction-tuned and reasoning models where the tail matters. Calibration-data quality influences the result more than the method — use data resembling your real traffic.
• Speed — quantized models serve faster, not slower, because inference is memory-bandwidth-bound: a 4-bit model moves half the bytes of an 8-bit model per token. A 70B at INT4 on 2× L40S often matches an FP16 70B on 1× H100 in tokens/s at a fraction of the hardware cost, trading some quality headroom.
• KV cache — quantize it (FP8 on Hopper, INT8 elsewhere) before buying extra GPUs; it is frequently the difference between "fits" and "does not fit" at long context.

Hardware Support — Which GPU Generation Runs What

This table is the real buying argument: if your roadmap is FP8 serving, Ampere cards (however cheap) are the wrong purchase — you pay for software-emulated FP8 or settle for INT8. Conversely, if you plan 4-bit GPTQ/AWQ, Ampere remains perfectly serviceable because those methods run in software on any CUDA GPU.

Practical Recommendations When You Buy

• Default stack for 2026 — FP8 with TensorRT-LLM or vLLM on Hopper/Ada for anything that fits at 8-bit; drop to 4-bit (AWQ preferred) only when VRAM or budget forces it.
• Ask your vendor for — a pre-baked quantized serving image (TensorRT-LLM engine + vLLM with GPTQ/AWQ/FP8 kernels compiled for your exact GPU), a benchmark of tokens/s at your context length, and burn-in with the quantized workload, not just a stock CUDA test.
• Size for the quantized model — state the target model and format in the RFQ. Vendors who quote "H100 for 70B" without asking about quantization are quoting you the 2023 answer.
• Plan for MoE — sparse models like DeepSeek-V3 load only active experts, so total VRAM matters less than bandwidth and expert locality — see the MoE requirements guide below.