When deploying a large language model, one of the most consequential decisions isn't which model you choose — it's which numerical format you run it in. Two models with identical parameter counts can require 4× different VRAM, run at very different speeds, and produce subtly different outputs. This guide breaks down the two most common formats: BF16 and Q4.
1. Bit Depth: How Numbers Are Stored
Every weight in a neural network is just a number. The precision format determines how many bits are used to represent that number — and what range and accuracy are possible.
BF16 matches FP32's exponent width (8 bits), so it preserves the same vast dynamic range — important for stable training and high-fidelity inference. The trade-off is that each weight consumes 2 bytes of memory.
Q4 compresses each weight into just 4 bits — half a byte — by mapping continuous floating-point values onto a lookup table of only 16 discrete levels. To partially compensate for this information loss, quantization tools (like GGUF/llama.cpp) apply per-block scaling factors that adjust the mapping across small groups of weights, preserving relative variance within each block.
2. VRAM Calculator
Memory footprint scales directly with parameter count and bits-per-weight. Use the calculator below to see the real-world difference for any model size.
⚙️ VRAM Estimator
Adjust the parameter count to estimate raw weight storage and operational VRAM requirements.
3. Accuracy, Perplexity, and Reasoning
Reducing from 16 bits to 4 bits is a lossy compression. Not all tasks are equally sensitive to this loss.
| Metric / Task | BF16 | Q4 |
|---|---|---|
| Perplexity | Baseline (reference) | +1–5% increase typical; varies by model family |
| Factual Q&A | Full fidelity | Negligible degradation in most benchmarks |
| Code generation | Full fidelity | Slight increase in syntax edge-case errors |
| Multi-step reasoning | Strongest | Noticeable drift in long chains of logic |
| Math / formatting | Deterministic precision | Higher propensity for subtle hallucination |
| Summarization / chat | Excellent | Excellent — nearly indistinguishable |
| Throughput (tokens/sec) | Lower on memory-limited hardware | Higher — less data to move per token |
LLM inference at small batch sizes is almost always memory-bandwidth bound, not compute bound. Q4 moves 4× less data per weight from VRAM to the compute units, so the GPU spends less time waiting for data — leading to higher token throughput even though the math itself is more complex.
4. Deployment Fit Guide
BF16 — Choose When
- Running on data-center GPUs (A100, H100, MI300)
- Fine-tuning or continued training
- Production APIs requiring maximum accuracy
- Complex agent pipelines with long reasoning chains
- Serving many concurrent users (batching benefits)
Q4 — Choose When
- Running on consumer GPUs (RTX 3060–4090)
- Local / offline deployment (no cloud)
- CPU inference or Apple Silicon via llama.cpp
- Edge devices, laptops, on-prem servers
- Chat and summarization tasks (quality is ≈ BF16)
For most local deployments, a well-quantized Q4 model is an excellent trade-off. The memory savings let you run a larger model in Q4 than you could in BF16 — and a larger Q4 model often outperforms a smaller BF16 one. When you have the hardware for BF16, use it for anything requiring rigorous accuracy. Otherwise, Q4 via GGUF is the pragmatic path.
5. Common Quantization Formats
Q4 is a family of approaches, not a single format. The most common implementations you'll encounter:
| Format | Description | Typical Use |
|---|---|---|
| GGUF Q4_K_M | 4-bit with K-quant mixed precision on sensitive layers | Best quality/size balance for llama.cpp |
| GGUF Q4_0 | Straight 4-bit, uniform quantization | Fastest load, smallest file |
| GPTQ Q4 | Post-training quantization with activation-aware calibration | GPU inference via ExLlamaV2 / vLLM |
| AWQ Q4 | Activation-aware weight quantization | Higher quality than GPTQ on many tasks |
| BitsAndBytes NF4 | 4-bit NormalFloat — used in QLoRA fine-tuning | Training on consumer hardware |