LLM VRAM calculator
How much GPU memory a model needs to run locally: weights by quantization, KV cache by context length, and whether it fits on your GPU. A planning estimate with the math shown, not a black box.
How the math works
Three things occupy VRAM during inference:
- Weights. Parameters times bytes per parameter. This part is exact arithmetic: a 70B model at FP16 is 140 GB, at 8-bit 70 GB, at 4-bit quantization 35 GB.
- KV cache. The attention keys and values for every token in context. It grows linearly with context length and with concurrent sessions, and at long contexts it rivals the weights: a 70B-class dense model at FP16 KV costs roughly 0.6 GB per 1,000 tokens of context. This is the number people forget, and it is why "the weights fit" is not the same as "the model runs".
- Overhead. Activations, runtime buffers, and the CUDA context. The calculator budgets 15%.
The KV figure assumes a typical dense transformer with grouped-query attention. Two families break the pattern in your favor and against it: DeepSeek and Kimi use multi-head latent attention, which compresses KV far below these estimates, while mixture-of-experts models must hold all parameters in VRAM even though only a fraction activate per token. Kimi K2's headline "32B active" still means roughly a trillion parameters resident: server hardware, not a desktop GPU.
VRAM requirements for a 70B model at Q4 quantization
The most-asked version of this question has a clean answer. A 70B dense model at 4-bit quantization needs about 35 GB for weights, plus KV cache, plus overhead. That means it does not fit a 24 GB RTX 4090, no matter the context length. It runs on a 48 GB card at short-to-moderate context, and comfortably on an 80 GB A100 or H100, where roughly 30 GB remains for context after weights and overhead, enough for tens of thousands of tokens.
What fits in your GPU
The gating question for "what is the best local model for my card" is what fits at all. Among the models that fit, pick by benchmarks on your actual task. What fits, by common card:
| VRAM | Card examples | What fits at Q4 (dense models) |
|---|---|---|
| 8 GB | RTX 3070, 4060 | 7-8B models (~4 GB weights) with modest context |
| 12 GB | RTX 3060 | 13-14B models (~7 GB weights) with moderate context |
| 16 GB | RTX 4060 Ti | 14B comfortably; 8B at 8-bit with long context |
| 24 GB | RTX 4090 | ~32B tightly (~16 GB weights); 14B with room for long context |
| 48 GB | RTX A6000 | 70B at Q4 with short context |
| 80 GB | A100, H100 | 70B at Q4 with long context, or ~120B-class tightly |
Derived from the weights formula above; verify against your runtime before buying hardware. Popular open-weight families in these size classes include Llama, Qwen, Mistral, and Gemma.
Why this lives on a token calculator site
Because it is the third leg of one decision. You count tokens to know your usage, you check API prices to know your bill, and when the bill grows, or a provider announces an increase, the next question is whether self-hosting an open-weight model beats paying per token. VRAM is the feasibility half of that answer. The two providers whose weights you can actually download, DeepSeek and Kimi, are covered here with the same rule as everything else: real numbers, labeled estimates, no black boxes.
Context length drives the KV term, so the context window comparison is the natural companion page.