How Much VRAM Do You Need for Local AI?

The most common local-AI buying mistake is treating parameter count as a VRAM requirement. A 70B model does not need 70GB of VRAM—NVIDIA's own deployment guidance for a 70B-class model puts the requirement at 140GB or more once precision, context, and runtime overhead are included. A 30B-class model quantized to INT4 can exceed 16GB once headroom is added, according to MSI's local-LLM planning tables.

The short answer: estimate weights first, then add context, batching, and overhead before comparing against a GPU's usable VRAM. Capacity decides what you can load. Memory bandwidth and software support decide how fast and how usable it is once loaded. Parameter count alone tells you almost nothing useful about either.

Why Parameter Count Is a Poor VRAM Predictor

A model's parameter count describes the weights—the learned values that make the model behave as it does. It does not describe the total memory footprint during inference. Weights must share VRAM with several other consumers:

• KV cache: memory holding the model's attention state as it processes your prompt and generates tokens
• Context window: the length of text the model can hold at once, which directly grows the KV cache
• Temporary activation buffers: intermediate values computed during generation
• Runtime overhead: the inference engine, tool, or application itself

The gap between raw weight size and real requirement is large enough to change your purchase. NVIDIA's cited example of a 70B-class model demanding 140GB VRAM or more comes from a specific blueprint context with particular precision and runtime assumptions—but it illustrates the point: weights are only the floor.

MSI's planning tables show the same effect at more accessible sizes. A 30B-class model at INT4 can exceed 16GB once headroom is added. A 13B-class model at INT8 can also exceed 16GB. If you were shopping by parameter count alone, you would expect both to fit comfortably in 16GB. They may not.

Quantization is the lever that shrinks the weight component. It reduces the precision of the weights—from FP16 to INT8 to INT4—roughly halving memory use at each step. The tradeoff is quality: lower precision can degrade output accuracy, though modern quantization methods have narrowed the gap considerably. You will use this lever constantly when fitting models to the VRAM you actually have.

The VRAM Budget: Weights, Context, and Overhead

Think of VRAM as a budget with three line items:

1. Model weights: the base cost, determined by parameter count and precision
2. KV cache and context: grows with context length and batch size, independent of model size
3. Runtime and application overhead: the inference engine and any surrounding tooling

The KV cache is where most estimates go wrong. Two people running the same model can need very different VRAM because one uses a 4K context and the other uses 32K. The longer the context, the more attention state the model must hold. Larger batch sizes—processing multiple prompts at once—grow the cache further. This is why a model that fits comfortably at short context can start failing at longer context on the same GPU.

Runtime overhead adds another layer. Some tools impose minimums that have nothing to do with the model itself. NVIDIA's ChatRTX, for example, requires 8GB VRAM regardless of which model you run. A laptop with a 6GB GPU can have plenty of CPU power and still be locked out of the tool entirely. When planning a purchase, check the tool's requirement, not just the model's.

A Quick Estimation Method

To turn a target model into a buying floor, work through these steps:

1. Find the actual file size of the model at your chosen quantization. A 7B model at INT4 might be roughly 4–5GB; at INT8, roughly 7–8GB. Use the real file size, not the parameter count.
2. Add context and batch growth. Longer context and larger batches expand the KV cache. A model that fits at 4K context can fail at 32K on the same GPU.
3. Add runtime overhead. The inference engine and any surrounding application consume VRAM beyond the model. Some tools impose fixed minimums.
4. Add headroom. Leave room for temporary buffers and for the fact that your usage will grow. A card that sits at 95% utilization with your current workload leaves no room for longer context or a larger batch.
5. Compare against usable VRAM, not nominal VRAM. On laptops especially, the GPU shares memory and power constraints with the rest of the system.

This is deliberately illustrative rather than a universal formula. Precision, runtime, context length, and overhead assumptions change every estimate. What matters is the structure: start with weights at your chosen quantization, add context and batch growth, then add runtime overhead, then leave headroom.

A Practical VRAM Ladder for Common Workloads

These tiers are capability illustrations, not universal recommendations. The evidence does not support a controlled cross-vendor matrix, and your specific model, quantization, and context will shift the boundaries. Treat each row as a conditional fit, not a guarantee.

VRAM tier	Likely fit	Conditional fit	Poor bet
8GB	Small quantized models, light experimentation	7B–8B-class models at higher precision with short context	Most tools with meaningful context; ChatRTX already requires 8GB minimum
12GB	7B–8B-class INT8, some 13B-class INT4 with short context	13B-class INT4 with longer context	30B-class models except at aggressive quantization with heavy offload
16GB	13B-class INT4 with moderate context, 7B–8B-class at higher precision	13B-class INT8, 30B-class INT4 with careful headroom management	Comfortable 30B-class work at longer context
32GB	30B-class INT4 with meaningful context, diffusion workloads	Larger quantized models with headroom	70B-class models except at aggressive quantization with offload
64GB–128GB unified	Very large models with shared memory	Models that exceed discrete VRAM entirely	Sustained speed on models that would fit in discrete VRAM

8GB tier: This is the entry floor for local AI, and it is already tight. Some tools require 8GB before loading any model, leaving nothing for the weights. Small quantized models and experimentation work, but you will hit walls quickly.

12GB tier: MSI's examples place 7B–8B-class INT8 and some 13B-class INT4 work here. This is a reasonable experimentation tier for someone who wants to try local AI without a major investment—provided you keep context modest.

16GB tier: This is where the arithmetic gets uncomfortable. MSI notes that 13B-class INT8 and 30B-class INT4 workloads can exceed 16GB once headroom is added. The RTX 5070 Ti and RTX 5080 both carry 16GB on desktop, and both can run out of memory on models that sound like they should fit.

32GB tier: The RTX 5090 at 32GB and AMD's Radeon AI PRO R9700 at 32GB both target this space. AMD positions its card for larger LLMs and text-to-image workloads. This tier gives you room for 30B-class INT4 work with meaningful context, plus headroom for diffusion models.

Larger unified-memory systems: Systems with 64GB to 128GB of shared memory can load very large models, but they come with different throughput and ecosystem tradeoffs. Capacity is not the only variable.

When System RAM and Offload Change the Answer

When weights exceed VRAM, you have a fallback: partial offload. Some layers run on the CPU using system RAM while others stay on the GPU. This lets a model load that would otherwise fail entirely.

The catch is speed. Offload trades capacity for token-generation rate. Layers running on the CPU are dramatically slower than layers on the GPU, and the constant movement between memory pools adds overhead. Offload is a capacity fallback, not a performance solution. It is the right choice when you occasionally need to run a model larger than your VRAM and can tolerate slow generation. It is the wrong choice when you need sustained speed on a model you run constantly.

Unified-memory systems take a different approach. On Apple silicon, AMD's Ryzen AI Max platforms, and NVIDIA's RTX Spark and DGX Spark systems, the GPU can address far more shared memory than a discrete card's VRAM. A Framework Desktop with AMD's Ryzen AI Max, for example, comes in 32GB, 64GB, and 128GB configurations, with community reports indicating up to 112GB addressable by the GPU.

The tradeoff is bandwidth. Unified memory offers large capacity but typically lower memory bandwidth than discrete GDDR7 VRAM. NVIDIA positions RTX Spark with up to 128GB of unified memory and claims it can run models up to 200B parameters—but that is a platform capacity claim, not independent validation across models, quantizations, and context lengths. The model may load. How fast it generates tokens is a separate question governed by memory bandwidth.

The decision rule: choose offload or unified memory when you need capacity for a large model occasionally. Choose discrete VRAM when you need sustained speed on models that fit.

Why Memory Bandwidth and Software Support Matter

VRAM capacity sets what fits. Memory bandwidth largely sets how fast tokens stream out during generation once the model is loaded. When a model is resident on the GPU and the runtime is well-supported, inference is memory-bound: the GPU must feed weights through the compute units continuously, and faster memory means faster generation.

The spread is wide. MSI lists the RTX 5080 at 960GB/sec and the RTX 5090 at 1,792GB/sec—nearly double. Unified-memory systems typically sit far lower, which is why a 128GB unified system can load a model a 32GB discrete card cannot, yet generate tokens more slowly on models both can run.

That said, bandwidth is not a standalone predictor. Prompt processing, offload, quantization, and software kernels can change which stage dominates. A card with high bandwidth but weak runtime support for your target model can lose to a card with lower bandwidth and better software. Once capacity is cleared, compare workload-specific throughput rather than buying on bandwidth alone.

Vendor-reported performance figures need context. NVIDIA reports 131 tokens per second for a 27B model on one RTX 5090 using multi-token prediction. That is a vendor-reported, workflow-specific result, not a general benchmark you should expect across models and settings. Treat it as evidence that the hardware is capable under favorable conditions, not as a guarantee.

Software ecosystem can matter more than raw capacity. CUDA, ROCm, and Metal are not interchangeable in practice. NVIDIA's CUDA ecosystem has the broadest model and tool support, which is why most local-AI guides default to it. AMD positions its Radeon AI PRO line with ROCm and PyTorch deployment, and support has improved, but not every model and tool runs identically on both. If your target model or toolchain has weak support on a particular vendor's stack, the extra VRAM will not save you.

The decision rule: capacity decides what you can load; bandwidth and software decide how usable and how fast it is once loaded.

Choosing a GPU: Decision Rules and Flip Points

The governing rule: buy the VRAM tier that fits your actual model, context, and batch, not the largest card you can afford. More VRAM is useful only when something real grows into it.

Move up a tier when:

• Your target model plus your typical context length exceeds current VRAM
• You need longer context or larger batches than your current card allows
• Offload speed becomes the recurring bottleneck in your workflow

Stay at a lower tier when:

• Your workload is small quantized models with short context
• You are experimenting and do not yet know what models you will run regularly
• The next tier buys headroom you have no plausible workload to fill

More VRAM is wasted when your workload never approaches the ceiling. A 32GB card running 7B-class INT4 models with short context is paying for capacity you will not use. The money would be better spent elsewhere in the system.

Watch the laptop-versus-desktop trap. The same GPU name can carry different VRAM on laptop and desktop. MSI notes that RTX 5070 laptops may have 8GB or 12GB, while the desktop version has 12GB. The RTX 5070 Ti laptop has 12GB, while the desktop has 16GB. If you are buying a laptop, verify the exact configuration rather than assuming the desktop spec carries over.

Sometimes the decision is not a GPU at all. If the bottleneck is software compatibility, more VRAM will not fix it. If you need occasional large-model capacity, a unified-memory system or offload may be the rational choice over an expensive discrete card you will rarely use at full capacity.

The Bottom Line

Estimate your VRAM floor from your actual workload: weights at your chosen quantization, plus context and batch growth, plus runtime overhead, plus headroom. Then compare that number to the GPU's usable VRAM, not to the model's parameter count.

Capacity decides what fits. Bandwidth decides how fast it runs. Software support decides whether it runs at all. Buy the tier that clears your real floor with useful headroom—and when the gap between what you need and what you can afford is wide, consider whether offload, unified memory, or a different toolchain changes the answer before you spend.