How to Choose Hardware for Local AI: VRAM, Memory, and Software Fit

If you're shopping for hardware to run local language models, vision models, or embedding pipelines, the first mistake is usually the same one: picking a GPU by brand or raw compute instead of by memory capacity. A faster card does you no good if the model you want to run won't fit in its VRAM. The model's weights, the KV cache, your context window, and runtime overhead all have to fit in memory before a single token gets generated. Compute speed only matters after that threshold is cleared.

This guide walks through how to translate the models you actually run into concrete hardware requirements, what VRAM versus unified memory versus system RAM really buys you, and when the rational move is quantization, more memory, a different platform, or no purchase at all.

What Actually Limits Local AI Hardware

Think of local AI hardware as having three gates, in order:

1. Memory capacity decides whether the model runs at all.
2. Compute and memory bandwidth decide how fast tokens generate once the model fits.
3. Software compatibility decides whether your chosen runtime and model work on the hardware at all.

Most buyers fixate on gate two because that's where the marketing lives. But gate one is the capability floor. A model whose complete workload needs more memory than your card provides simply will not run at full speed on that card, regardless of how many CUDA cores or tensor operations it claims. And a platform with plenty of memory but poor support for your inference tool may refuse to run the model entirely.

The distinction between memory types matters here. Dedicated VRAM on a discrete GPU is fast and well-supported but capped by the card's physical capacity. Unified memory, where the CPU and GPU share one pool, allows much larger models but shares bandwidth across everything. System RAM can hold model data that spills past VRAM, but offloading to it is slow enough that it should be treated as a stopgap, not a solution.

A smaller discrete GPU with a mature software stack can beat a larger unified-memory system for a workload that fits comfortably. The reverse is also true: a unified-memory system with 128GB can run models that no consumer discrete GPU can touch. Neither approach wins outright. The workload decides.

Start With the Model, Not the GPU

Before comparing spec sheets, write down the models you want to run daily, not the largest model you can imagine trying once. Then estimate what each one needs.

The rough starting point for memory is the parameter count multiplied by bytes per weight at your chosen precision. A 7-billion-parameter model at 4-bit quantization needs roughly 3.5GB just for weights. The same model at 8-bit needs about 7GB. A 13B model at 4-bit lands around 6.5GB. These are planning heuristics, not guarantees: the actual footprint depends on the model architecture, the runtime, and how memory is allocated.

That weight estimate is only the floor. You also need room for the KV cache, which grows with context length and batch size, not just model size. A long-context session can add several gigabytes on top of the weights. Runtime overhead and the application itself consume more. A practical planning rule is to add 20–30 percent headroom to your weight estimate, then verify the total against the actual model and runtime before you commit to a purchase.

Quantization is the lever that changes this math. Dropping from 8-bit to 4-bit roughly halves the weight footprint at some cost to output quality. For many coding-assistant and embedding workloads, that tradeoff is invisible in practice. For tasks where precision matters, the quality loss may be unacceptable. Test before you buy, because quantization is the cheapest way to make a model fit on hardware you already own.

Model class also changes the profile. Language models are memory-hungry and benefit from fast token generation. Vision models add compute pressure for image encoding. Embedding models are small enough that most modern hardware handles them easily, so buying a workstation for embeddings alone is usually overkill. Match the hardware tier to the heaviest workload you run repeatedly, not the one you tried once.

VRAM, Unified Memory, and System RAM: What Each Buys You

The three memory types produce very different buying decisions.

Dedicated VRAM on discrete GPUs is the most straightforward path for local AI. NVIDIA positions GeForce RTX systems with roughly 6–32GB of VRAM for developing and testing smaller models, with RTX PRO systems spanning a 16–96GB range for larger development work. The advantage is maturity: CUDA support is broad, and tools like Ollama, llama.cpp, TensorRT, and PyTorch work with minimal friction. The limit is capacity. If your target model needs more than the card holds, you are done unless you quantize harder, offload to system RAM, or split across multiple GPUs.

Unified memory lets the CPU and GPU share one pool, which changes what can fit. The Framework Desktop built around AMD's Ryzen AI Max APU comes in 32GB, 64GB, and 128GB configurations, and the 128GB tier can hold models that no discrete consumer GPU approaches. NVIDIA's RTX Spark and DGX Spark systems similarly offer up to 128GB of unified memory, with vendor positioning for models up to the 200-billion-parameter class. Apple Silicon uses the same principle.

The catch is that unified memory shares bandwidth between the CPU and GPU. A large pool can fit a model that still generates tokens slowly because bandwidth or compute becomes the limit. Framework's own community documentation notes that inference ran about 20 percent faster on Fedora 42 than on Windows 11 as of mid-2025 — a reminder that the software stack changes the result as much as the silicon does, and that this kind of observation is platform-specific, not a universal OS rule.

System RAM matters when models spill past VRAM. CPU offload lets you run a model that technically exceeds your GPU's capacity, but the speed penalty is severe because every layer that leaves the GPU crosses the PCIe bus. Treat offload as a way to experiment, not a way to work.

Before buying, check whether memory is user-upgradeable or fixed. A discrete GPU can be swapped later. Unified-memory systems often cannot be expanded at all, which means the configuration you buy sets your model ceiling for the life of the machine.

Compute, Bandwidth, and Sustained Speed

Once the model fits, speed depends on compute throughput and memory bandwidth working together. Token generation is memory-bound for many models, which is why a card with modest compute but wide memory bandwidth can outperform a compute-heavy card with narrow bandwidth. But bandwidth is not everything. Professional reviews comparing Apple's M4 Max Mac Studio against other unified-memory platforms have explicitly cautioned that memory bandwidth alone does not predict real inference throughput. The complete workflow — runtime, quantization, model implementation, and operating system — decides the result.

Sustained performance matters more than peak. A laptop GPU can hit impressive clocks for a few minutes, then throttle as the chassis heats up. A desktop card with adequate cooling holds higher clocks for the duration of a long session. If you run code-assistant models for hours, the system that sustains its performance wins regardless of what the spec sheet claims at idle.

Vendor-reported throughput figures deserve skepticism. NVIDIA's example of a GeForce RTX 5090 running a specific model with a specific software feature is context-bound, not a universal result across models and cards. Treat any single-number performance claim as a data point from one configuration, then look for tests that match your actual workload.

The practical frame: buy for the model you run repeatedly, not the largest model you can imagine running once. A system that generates tokens at a comfortable rate for your daily coding assistant is worth more than one that can technically load a massive model but produces output too slowly to use.

Software Fit: CUDA, ROCm, Metal, and the Runtime Stack

Software compatibility can override raw hardware capability. The runtime and backend decide whether your model runs at all, and the ecosystems differ sharply.

NVIDIA's CUDA ecosystem has the broadest documented support across the major frameworks and runtimes. Ollama, llama.cpp, TensorRT, vLLM, and PyTorch all list CUDA support, which makes GeForce and RTX PRO systems the lowest-setup-risk choice for developers who want to minimize friction between downloading a model and running it. That is a compatibility argument, not a universal performance verdict.

AMD's ROCm platform and Ryzen AI Max systems can run local models, and AMD's own guidance shows a 128GB configuration running a 106-billion-parameter model on Windows. But the practical friction varies by tool and model. Some runtimes have mature AMD support; others lag behind their CUDA paths or require extra configuration. The capability is real; the convenience is not always equal. Check your exact runtime, model, and quantization before assuming parity.

Apple Silicon uses Metal and its own stack. It is strong for certain workloads, but you must verify that your specific runtime and model support it before buying. Some tools are Linux-first or CUDA-first, and the Mac version may be missing features or run slower.

Operating system matters on the same hardware. The Framework community's observation that Linux ran inference faster than Windows on the same machine is not an isolated pattern. If you plan to run Linux, check that your GPU and runtime are well-supported there. If you plan to run Windows, accept that some tools may behave differently.

Check driver and runtime support for your exact model and quantization before purchase, not after. A platform that runs most models well but not the one you use daily is the wrong platform.

Storage, Cooling, and the System Around the GPU

The components around the GPU become the real bottleneck when ignored.

Models and datasets are large. A single model file can be tens of gigabytes, and if you experiment with several, storage fills quickly. Fast NVMe storage for model files and checkpoints matters more than most buyers expect, both for load times and for the ability to switch models without waiting.

Cooling and power delivery determine whether the GPU sustains its rated performance. A card that throttles after twenty minutes of inference is effectively a slower card. Compact systems and laptops trade sustained performance for size; a deskside box with proper cooling holds higher sustained clocks. If your sessions run long, read reviews that measure sustained throughput rather than peak benchmarks.

Multi-GPU scaling adds PCIe, power, cooling, and software-coordination constraints that most single-GPU buyers should not assume away. Splitting a model across two cards sounds appealing until you deal with the bandwidth between them and the runtime support required. For most buyers, one card with enough memory beats two cards that each hold half the model.

The total ownership burden includes power draw, noise, and physical space. A system that generates a constant hum while running inference may be fine in an office and unacceptable in a bedroom. A dedicated local-AI box like an RTX Spark or DGX Spark makes sense when you need inference isolated from your main machine or available around the clock — and you accept that it is another system to power, update, and maintain. If your usage is intermittent or the machine must double as your daily workstation, a shared system or a cloud fallback is often the more rational choice.

Minimum, Recommended, and Diminishing-Return Tiers

A practical tier ladder looks like this. Treat the model sizes as illustrative starting points, not guarantees: usable headroom and tokens-per-second depend on the exact model, quantization, context length, and runtime.

Tier	What it handles	Who should buy	Who should skip
Minimum viable	Small quantized models (roughly 7B–13B class) on a discrete GPU with 8–16GB VRAM, or a modest unified-memory system	Experimenters, light coding assistants, embedding workloads	Anyone running large models or long contexts daily
Recommended	Models up to roughly the 30B class quantized with workable context, or larger models on 64GB-class unified memory	Developers running coding assistants and local agents regularly	Anyone whose daily model fits in the minimum tier
Diminishing returns	100B+ parameter models, long-context workloads, multi-model experimentation	Users who repeatedly run large models and need the capacity	Anyone whose workload never grows into the extra memory

NVIDIA's product families illustrate these tiers: GeForce RTX systems for smaller models, RTX PRO for larger development, and RTX Spark or DGX Spark for the 200B-class unified-memory tier. AMD's Ryzen AI Max configurations cover similar ground from the unified-memory side. These are examples of capability classes, not endorsements. The decision boundary is the same across brands: pay more only when a real workload grows into the extra capacity. Otherwise, the base tier delivers the same repeated result.

Common Buying Mistakes and How to Avoid Them

The recurring errors are consistent enough to name:

Buying the fastest GPU without checking model fit. Capacity gates capability before speed matters. A 24GB card that runs your model is more useful than a 32GB card that cannot run the next model up.

Assuming unified memory equals discrete-GPU performance. A large unified pool can fit a model that still generates tokens slowly. Capacity and speed are separate questions.

Ignoring software compatibility. Buying a platform your runtime and model do not support well guarantees frustration regardless of the hardware's theoretical capability.

Overbuying or underbuying memory. Buying 128GB for workloads that never exceed 16GB wastes money. Buying 16GB when you plan to run a 30B model daily wastes more.

Treating vendor model-capacity claims as speed guarantees. NVIDIA's "up to 200B" positioning for RTX Spark and DGX Spark describes a fit ceiling, not a promise of usable speed at that size with your context length and quantization.

Forgetting the system around the GPU. Storage, cooling, power, and OS can become the real bottleneck. A great GPU in a poorly cooled chassis underperforms a modest GPU in a well-designed system.

Decision Rule: When to Upgrade, Add RAM, Quantize, or Change Workflow

When your current system falls short, work through the levers in order:

If the model does not fit in memory: Quantize first. Dropping from 8-bit to 4-bit is free capacity if the quality loss is acceptable. If quantization is not enough, then consider more VRAM or unified memory. If neither is practical, a smaller model that fits comfortably beats a larger model that runs at unusable speed.

If the model fits but generation is slow: The bottleneck is compute or bandwidth, so more memory will not help. A faster GPU or a smaller model is the lever. Measure tokens per second with your actual model and context before spending.

If the software stack does not support your hardware: No amount of memory or compute fixes this. Change platform or runtime. This is the one case where a hardware purchase is the wrong answer entirely.

If thermals throttle long sessions: Cooling, power delivery, or a desktop form factor is the fix, not a bigger GPU. A system that sustains its clocks will outperform one that peaks and drops.

A new purchase is justified only when you have a workload you run repeatedly that the current system cannot fit or sustain. Not a model you want to try once. Not a benchmark that looks impressive. A recurring job that the current hardware fails.

Start with a concrete estimate: pick your target model, note the parameter count and quantization, add context overhead, and write down the memory requirement. Then check that your runtime and drivers support the model on the platform you are considering. Only after those two checks should you compare systems on price. Fit first, speed second, and the brand badge somewhere far behind both.