VRAM Requirements for Running LLMs Locally: A Complete Guide
When deploying a local LLM, the primary concern is whether it can fit within your GPU’s memory. The solution hinges on three key factors: model size, quantization level, and context length. This guide provides a practical framework for determining the optimal VRAM capacity for your needs.
The Impact of Quantization on VRAM
Quantization lowers the precision required to store model weights. While lower-bit quantization reduces the model’s footprint and VRAM consumption, it often comes with a trade-off in output quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good quality and size |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM, more quality loss |
If VRAM is a constraint, Q4_K_M is frequently the preferred choice. However, with more available memory, you can opt for Q5 or Q6 to preserve higher quality with less aggressive quantization.
Estimated VRAM Needs by Model Size
The following figures provide rough estimates for model weights alone. In practice, total VRAM consumption will be higher due to runtime overhead, KV cache, and context requirements.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
Note that these are approximations rather than strict limits. Variations in model architecture and quantization formats can influence the actual memory footprint.
Model Capabilities by VRAM Tier
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models around 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models around 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges assume the model weights can reside entirely on the GPU. MoE (Mixture-of-Experts) models behave differently: while only a subset of parameters is active per token, the system must still store the full weight set. Consequently, a model with 100B or more total parameters will not fit into a VRAM budget matched to its active parameter count alone.
Understanding MoE Models
Mixture-of-Experts models utilize multiple parameter groups, or 'experts'. Since only specific experts are activated for each token, inference can be more efficient than dense models of the same total size.
However, the inactive experts still occupy memory. As a result, large MoE models may demand significantly more memory than their active parameter count suggests. Very large configurations may necessitate multiple GPUs or offloading to system RAM.
Context Length and VRAM Consumption
Model weights represent only a portion of total memory usage. The KV cache expands with context length, meaning a 64K context requires substantially more VRAM than a 4K context for the same model.
- Extended contexts consume additional VRAM.
- KV-cache precision influences memory requirements.
- Batch size and concurrent user load increase memory usage.
- Reserve sufficient VRAM for the runtime, avoiding filling the GPU entirely with model weights.
Practical Recommendations
- Verify the actual size of the specific quantized model you intend to use.
- Do not rely solely on file size as the exact VRAM requirement; account for KV cache and runtime overhead.
- If the model exceeds VRAM capacity, offloading part of it to system RAM is possible, though it typically reduces inference speed.
- For long-context or agentic tasks, allocate more VRAM than required for the weights alone.
- Utilize multiple GPUs to distribute the model if a single GPU lacks sufficient VRAM.
Launch on DaDesktop
There is no need to purchase physical hardware to run local LLMs. DaDesktop provides cloud-based desktops with the necessary VRAM, allowing you to deploy models directly without owning the underlying infrastructure.
Select a VRAM tier that matches your model, load it, and begin working. Enjoy a seamless experience with no setup, hardware purchase, or driver complications. View available GPUs for detailed options.