Every model you deploy needs somewhere to live, and "somewhere" means GPU VRAM. Undersize it and inference crashes with an out-of-memory error the moment a second user connects. Oversize it and you're paying for silicon you don't need. The two variables that actually drive the number — concurrent users and context length — are the ones people forget, because model weights alone look deceptively small on a spec sheet.
1. The Core Equation
VRAM demand splits into two pools: a fixed cost for the model's weights, and a variable cost for the KV cache — the per-token memory each active conversation reserves to avoid recomputing attention from scratch.
P = Parameters (billions) · B = Bytes per parameter (precision) · U = Concurrent users · T = Context length (tokens) · C = KV cache constant
The result is then multiplied by 1.2× — a 20% buffer that covers CUDA kernels, activation tensors, and other runtime overhead that isn't captured by the weight or cache math but will absolutely show up in nvidia-smi.
2. Precision Reference
Bytes-per-parameter is set by how aggressively the model is quantized. Lower precision shrinks the weight pool linearly, at some cost to output quality — see the Model Precision guide for the accuracy trade-offs.
FP16 — 2 bytes
Full quality, highest VRAM. Standard for production-grade serving.
INT8 — 1 byte
Half the VRAM of FP16 with minimal measurable quality loss.
INT4 — 0.5 bytes
Aggressive quantization for consumer hardware and edge deployment.
3. Interactive VRAM Calculator
Adjust the model size, precision, concurrent user count, and context length to see the required VRAM and a matching hardware recommendation.
🖥️ GPU VRAM Sizing Tool
Estimates video memory for LLM inference, including KV cache overhead.
4. Hardware Tiers
Once you have a total VRAM figure, match it to the smallest tier that clears it with room to spare — undersizing by even a few GB will surface as intermittent OOM errors under load.
| Total VRAM | Recommended Hardware |
|---|---|
| ≤ 24 GB | RTX 4090 / RTX 5080 (24GB) — single-user or light dev workloads |
| ≤ 48 GB | RTX 6000 Ada / L40S (48GB) — small team serving, moderate context |
| ≤ 80 GB | A100 / H100 (80GB) — production single-GPU serving |
| > 80 GB | Multi-GPU / H200 — tensor-parallel serving required |
KV cache scales with users × tokens, not just model size — every concurrent conversation reserves its own cache for every token in its context. Doubling either concurrent users or context length roughly doubles the cache pool, which is why chat products with long context windows need far more headroom than a single-user coding assistant.
Size for peak concurrency, not average load — VRAM can't be borrowed mid-request. If your estimate lands close to a tier boundary, round up: the 20% buffer in the formula covers steady-state overhead, not traffic spikes or larger-than-expected prompts.