🚀 Local LLM Memory Requirements: 2026 VRAM & KV Cache
Learn local LLM memory requirements for 2026: calculate LLM VRAM needs, KV cache memory, quantization impact & runtime overhead to run LLM locally.
Will a Local LLM Fit? A Practical Memory Sizing Guide
Choosing a local language model is easy - right up until you try to load it. The advertised parameter count doesn't tell you whether it will fit in GPU memory or system RAM, and the download size alone can be misleading. The memory you need shifts with quantization, context length, runtime overhead, and the hardware you're actually using.
This matters because a model that "loads" can still be a hassle. If it runs out of memory and falls back to disk, generation can crawl. If you pick a model that's bigger than your machine can comfortably handle, you may end up sacrificing quality or capability unnecessarily.
A practical estimate breaks the problem into three parts: model weights, the key-value (KV) cache, and runtime overhead. Add those up, then compare the total to the memory your hardware can realistically provide - not the number printed on the spec sheet. This guide walks through the calculation, shows what it means for a 14B model on a 16 GB Mac, and explains how memory estimates can also hint at generation speed.
How much memory does a local LLM need?
There isn't a single "the model uses X GB" number. What you need depends on the model file, how much context you ask it to handle, and the runtime that loads it. A useful first-pass estimate is:
Memory needed = weights + KV cache + runtime overhead
This estimate only helps if you compare it to the memory you can actually make available to the model. As a rule of thumb, treat a model as a comfortable fit when it uses no more than about 80% of usable memory. That buffer covers the operating system, other apps, and the fact that runtimes account for memory a little differently.
Why the download size is not the whole answer
Weights are usually the biggest chunk, so file size is a decent starting point. But your setup still needs memory while the model is generating. The KV cache holds information for tokens the model has already processed, and it grows as your prompt (or conversation history) gets longer. On top of that, the runtime needs working buffers for computation.
So "the model file is 8 GB and I have 8 GB of memory" is not a safe fit check. The process needs more than the file itself, and your operating system is already using memory.
A worked example: Qwen2.5 14B in Q4_K_M
For a 14-billion-parameter model in Q4_K_M quantization, the weights are roughly 7.8 billion bytes, or about 7.3 GiB. Add a 4K-context FP16 KV cache of about 768 MiB and runtime overhead of roughly 0.9 GB, and you land around 8-9 GB total. The exact number depends on how the app reports units and what buffers it counts.
One local-model estimator, for example, reports about 8.4 GB for this configuration. That figure is for the stated model, context, and assumptions - it's not automatically true for every 14B model. Different architectures, runtimes, context settings, and memory accounting can change the result.
The takeaway is simple: estimate the full workload, not just the parameter count. A 14B model might be fine at one context length and quantization, but get painful if you increase context or switch to a different compression level.
What determines an LLM's memory use?
The three pieces in the estimate behave differently. Weights are driven mostly by parameter count and quantization. KV cache depends on the model architecture, cache precision, and context length. Runtime overhead varies with the inference engine and how you configure it.
Weights and quantization
A basic way to estimate weights is:
Number of parameters × bytes per weight
The bytes per weight depend on the numerical format. Approximate values for common formats are:
| Quantization | Approximate bytes per weight |
|---|---|
| FP16 / BF16 | 2.0 |
| Q8_0 | 1.0 |
| Q6_K | 0.82 |
| Q5_K_M | 0.69 |
| Q4_K_M | 0.56 |
| Q3_K_M | 0.44 |
| Q2_K | 0.31 |
These are estimates, not exact "guaranteed file size" numbers. For example, Q4_K_M uses about 0.56 bytes per weight rather than exactly 0.5, because the format stores extra scaling information for blocks of weights. In practice, the effective size is closer to 4.5 bits per weight.
For a 14B model, multiplying 14 billion parameters by 0.56 bytes gives about 7.84 billion bytes, or around 7.3 GiB. If you don't know the quantization, assuming FP16 is a conservative option. It may exclude some models that would otherwise fit in a compressed form, but it avoids treating an unknown configuration as automatically safe.
KV cache and context length
During generation, the model keeps keys and values for tokens it has in its current context. A simplified FP16 cache estimate is:
2 × layers × KV heads × head dimension × context tokens × bytes per element
The factor of two accounts for keys and values. For Qwen2.5 14B, using 48 layers, 8 KV heads, a head dimension of 128, and a two-byte FP16 cache, the estimate is about 768 MiB at 4K context. At 32K context, it rises to roughly 6 GiB.
That jump is the big reason context settings can make an otherwise "good fit" turn into a memory problem. A model might run comfortably with short prompts, then run out of room when you load a long document or keep a large conversation history. Some runtimes also support different cache optimizations or precisions, which can reduce KV cache use - but the exact outcome depends on both the model and the software.
Runtime overhead
Inference software needs extra working memory on top of weights and cache. A rough planning number is:
- about 10% of the weight size, plus
- roughly 200 MB for runtime buffers
For the 14B Q4_K_M example, that puts overhead around 0.9 GB.
This is deliberately a rule of thumb. Real overhead depends on the inference engine, hardware backend, batching, and configuration. Use it as a planning allowance, then confirm using the runtime's own estimate or by monitoring memory in a representative test. Also remember the important point: not all of your available memory can be assumed to go directly to model weights.
How do you estimate usable memory on your hardware?
The RAM your computer advertises and the memory you can actually use for an LLM aren't always the same thing. The main difference is whether the model runs on a dedicated GPU or on shared memory (like Apple Silicon).
Dedicated GPU: use VRAM as the limit
With a dedicated GPU, the practical ceiling is the GPU's VRAM. A 12 GB graphics card doesn't become a 24 GB card just because your machine has 32 GB of system RAM. Some runtimes can offload parts of a model to system memory, but that usually hurts performance and adds complexity.
For a first estimate, use VRAM - not total system RAM.
Then compare your total requirement against that limit. A model estimated at 10 GB on a 12 GB GPU might load, but it leaves little headroom for other GPU work or for how the runtime allocates buffers. If your estimate is above 12 GB, don't treat it as a clean "load everything on VRAM" situation.
Apple Silicon: account for shared memory
Apple Silicon uses unified memory shared by the CPU and GPU. That's more flexible than having separate VRAM, but the GPU still can't safely consume all installed RAM - macOS and other applications need memory too.
A reasonable rough planning approach is to assume you can use about 75% of installed memory. On a 16 GB Mac, that's around 12 GB. It isn't a fixed law - actual headroom changes with system activity and your workload. Still, it's useful because it stops the common mistake of assuming all 16 GB is available to the model.
Apply a fit rule, not a binary guess
Once you have a total requirement and an estimate for usable memory, classify the result:
- At or below 80% of usable memory: generally comfortable, with room for the system and other apps.
- Between 80% and 100%: tight. It may load, but you should close other apps and test the context length you actually want.
- Above 100%: expect memory pressure or offloading. If the system starts swapping to disk, interactive generation can become painfully slow.
For the 16 GB Mac example, a Qwen2.5 14B estimate of 8.4 GB is about 70% of the 12 GB planning budget, so it's a reasonable fit. Q5_K_M at 10.4 GB uses about 87% and is tight. Q6_K at 12.3 GB and Q8_0 at 14.9 GB exceed the estimated usable amount.
You can use the same method to shortlist models before you download them. In one comparison using Q4_K_M, the boundary fell between a 20B model at 11.9 GB (tight) and a 27B model at 15.9 GB (over budget). The exact boundary will vary with model, context, and runtime - use it as a filter, then verify with the configuration you plan to run.
What does memory use tell you about speed?
Memory sizing can also give a rough expectation for generation speed. During token generation, the model repeatedly reads its weights from memory. That makes memory bandwidth a major limiter, especially for single-user inference where the model generates one token at a time.
A simple upper bound is:
Maximum generation rate ≈ memory bandwidth ÷ model size
If an M3 provides around 100 GB/s of memory bandwidth, dividing by an 8.4 GB model gives a theoretical ceiling of roughly 12 tokens per second. An 8B model estimated at 4.9 GB would have an upper bound near 20 tokens per second on the same bandwidth.
Treat the estimate as a comparison, not a promise
Real generation rates are lower than this calculation. Runtime efficiency, model architecture, quantization, hardware utilization, context size, and prompt-processing work all affect measured performance. The formula is most useful when comparing models on the same machine: if two models run under similar conditions, the smaller one often generates faster because there are fewer weight bytes to read per token.
Also separate generation speed from prompt processing. The estimate above covers the repeated work involved in producing output tokens. Processing a long prompt has different performance characteristics and can take noticeably longer - especially with large contexts.
Why this matters when choosing a model
A model that barely fits can be slow, but "can it fit?" and "how fast is it?" are different questions. Bandwidth affects how quickly data moves; capacity decides whether the workload fits without costly fallback behavior. You can have enough memory but modest bandwidth, or high bandwidth but insufficient capacity for a large model and long context.
For practical selection, start by eliminating setups that exceed your memory budget. Then compare what remains for quality, context needs, and measured speed. If you want a local coding assistant, for example, a smaller model that responds reliably can be more useful than a bigger one that only works with a short context or has sluggish interactive latency.
Key takeaways
- Estimate weights, KV cache, and runtime overhead together.
- Include quantization and context length; without them, memory estimates are incomplete.
- Compare the total against usable VRAM or a conservative shared-memory budget - not installed RAM alone.
- Keep headroom. "It loads" isn't the same as "it's comfortable."
- Use the memory-to-bandwidth calculation to compare likely generation speed, then benchmark your actual workload.
Conclusion
If you want to know whether a local LLM will run well, don't stop at parameter count or download size. Estimate weights, add KV cache for your intended context, and include a realistic runtime allowance. Compare that total to usable memory, leaving enough headroom for the operating system and the rest of your workload.
This approach answers two questions before you sink time into downloading and configuring a model: whether it's likely to fit, and how its size may affect generation speed. Use the calculation to narrow your options, then test the exact quantization, context, and runtime you plan to use. A model that fits comfortably and behaves consistently is usually the better choice than one that only works on paper.