How Much System RAM Does Local AI Need? Offload, Context, and Bottlenecks

A familiar local-AI story: someone builds a machine with 128GB of system RAM, expecting it to run a 70B-parameter model at useful speed. The model loads. Then tokens trickle out at a fraction of the expected rate, or the system starts swapping to disk and grinds to a halt.

The mistake is treating system RAM as if it were VRAM. It isn't. The model's location—not the total memory on the motherboard—determines both how fast inference runs and how much system memory you actually need.

Before you spend money on more RAM, a bigger GPU, or a different platform, answer one question: what is actually limiting your workload? This guide walks through how model placement, context length, and the KV cache drive system-memory demand, then shows you how to identify whether RAM capacity, VRAM, memory bandwidth, or raw compute is the real bottleneck.

Why System RAM Is Not a Substitute for VRAM

A model fully resident in VRAM barely touches system RAM during inference. The weights live on the GPU, compute happens on the GPU, and system memory sits mostly idle. If that describes your workload, adding system RAM does nothing.

The confusion starts with discrete-GPU systems, where VRAM and system RAM are separate pools connected by a relatively slow bus. A GPU with 16GB of VRAM cannot reach into 64GB of system RAM and use it as if it were video memory. When a model exceeds VRAM, the runtime offloads layers to system RAM, and every offloaded layer crosses that bus during inference. That crossing is expensive.

Unified-memory systems work differently. On Apple Silicon, AMD's Ryzen AI Max line, or NVIDIA's RTX Spark and DGX Spark class of machines, one memory pool serves both CPU and GPU. The GPU can address the full capacity without copying across a bus. That changes what "system RAM" means for AI work, but it does not make capacity equivalent to speed.

There is also a gap between nominal and usable capacity. The operating system, inference backend, tokenizer, and any running applications reserve system memory before the model gets anything. Windows 11 lists 4GB as a minimum requirement, and real-world use with a browser or editor eats far more. A 32GB machine is not a 32GB machine for your model; it is 32GB minus everything else running.

That leaves a placement spectrum with three practical positions:

• Fully in VRAM: The model never touches system RAM during inference. Fastest option, but capacity-limited by the GPU.
• Partially offloaded: Some layers spill to system RAM when the model exceeds VRAM. Slower, but lets you run models that would not otherwise fit.
• Fully CPU-resident: The model runs entirely from system RAM using CPU compute. Slowest option, but with no VRAM ceiling.

Your system-RAM requirement depends entirely on where your model lands on that spectrum.

What Actually Drives System-RAM Demand

Four components consume memory during local inference. Understanding each lets you estimate requirements instead of guessing.

Model weights. Base size is the number of parameters multiplied by bytes per parameter at your chosen precision. A 70B model at 4-bit quantization needs roughly 35GB for weights alone. At 8-bit, that doubles to roughly 70GB. Precision directly scales the weight footprint.

KV cache. This is the component most people underestimate. The KV cache stores key-value pairs from the attention mechanism for every token in the context window. It grows with context length and batch size, and it is allocated in addition to model weights. A model that fits comfortably in VRAM at 4K context may not fit at 128K context, and the overflow has to go somewhere.

Offloaded layers. When a model exceeds VRAM, the runtime spills layers to system RAM. The spill amount sets your system-RAM floor. A Q4 70B model needs roughly 35GB; on a 24GB GPU, at least 11GB lands in system RAM before you account for the KV cache and everything else.

Runtime, OS overhead, and concurrency. The inference backend, tokenizer, and operating system each reserve memory before the model loads. Running multiple model instances, agent workflows, or a model alongside other applications raises the practical floor. Size for the combined load, not the model alone.

NVIDIA's own guidance illustrates how far model size can outrun a single consumer GPU. In its technical blog on choosing a first local AI project, NVIDIA states that a 70B model in its cited configuration demands roughly 140GB of VRAM or more, while an 8B alternative has substantially lower memory requirements. That figure is specific to NVIDIA's example configuration—quantization, context length, and runtime all shift the number—but it makes the point: a 70B model is not a single-consumer-GPU workload, and no amount of system RAM changes that.

Estimating Your Requirement: A Worked Method

You can estimate your system-RAM requirement in four steps. The goal is a floor to verify, not a precise spec.

Step 1: Pick the model and quantization you plan to run. Look up its approximate memory footprint at that precision. Model cards and runtime documentation usually list this. A Q4 8B model needs roughly 4–6GB for weights. A Q4 70B model needs roughly 35–40GB.

Step 2: Add KV-cache allowance for your intended context length. Longer context and multi-turn sessions increase this. If you plan 32K context on a larger model, the KV cache can add several gigabytes on top of the weights. Batch inference or agent workflows that process multiple requests add more.

Step 3: Decide placement. A model fully resident in VRAM needs almost no system RAM beyond the OS and runtime. A partially offloaded model needs system RAM for the offloaded layers plus whatever KV cache spills. A CPU-only model needs everything in system RAM.

Step 4: Add a working margin. The OS, runtime, and concurrent applications consume memory before the model gets anything. A 4–8GB margin is reasonable for a dedicated inference box; more if you plan to work on the machine while the model runs.

Here is a realistic example. An 8B model at Q4 fits comfortably in a 16GB GPU. The model lives entirely in VRAM, system-RAM pressure stays minimal, and a 16GB or 32GB system is plenty. Now consider a 70B model at Q4 on that same 16GB GPU. The model needs roughly 35GB, so more than half must offload to system RAM. Your system-RAM floor jumps to roughly 20GB or more before the KV cache, OS, and everything else. A 32GB machine may load it, but only with a short context, a lean desktop, and no concurrent work. A 64GB machine gives you room to breathe.

That last distinction matters. Loading a model and running it comfortably are different thresholds. Capacity to load means the weights fit in memory. No-paging headroom means the OS, runtime, and concurrent apps also fit. Acceptable speed is a separate question governed by bandwidth and compute, not capacity. A model that loads but generates at 2 tokens per second has not failed to fit; it has failed to perform.

Exact footprints vary by runtime, quantization scheme, and model architecture. Treat your estimate as a starting point, then verify with a memory monitor while the model runs. Most runtimes show VRAM and system-RAM utilization alongside token-generation rate.

When CPU Offload Makes Sense and When It Does Not

Offload lets a model larger than your VRAM still run. The runtime keeps some layers in system RAM and swaps them in as needed. It is a genuine capability, and for many users it is the difference between running a model and not running it at all.

The cost is speed, and the cost can be severe. System-RAM bandwidth is far lower than VRAM bandwidth or the high-bandwidth unified memory on integrated platforms. Every offloaded layer must cross the bus during inference, and that crossing becomes the bottleneck. A model that generates 40 tokens per second fully in VRAM might drop to 5–10 tokens per second with significant offload. Interactive use becomes painful.

The Framework community's testing on the Ryzen AI Max+ 395 illustrates the bandwidth dimension. Framework's own material cites 256GB/sec of shared memory bandwidth for the platform, and community testing on pre-production Framework Desktop hardware shows that even with that relatively high bandwidth, a dense 70B model generates only a few tokens per second while a smaller model runs much faster. That figure is specific to that platform and should not be generalized to all systems, but it demonstrates the governing principle: bandwidth, not just capacity, shapes the result.

The decision rule is straightforward. Offload is reasonable when speed is secondary—batch processing, overnight jobs, or experimentation where a slow response is acceptable. It is not a substitute for a GPU that fits the model when you need responsive interactive use. A large system-RAM pool does not make a model equivalent to a VRAM-resident workload.

Offload is also distinct from unified memory. On a discrete-GPU system, offload crosses a slower bus between separate memory pools. On a unified-memory system, the GPU addresses one high-bandwidth pool directly. The two architectures have different performance characteristics, and the unified-memory comparison covers that tradeoff in full.

Reading the Real Bottleneck: RAM, VRAM, Bandwidth, or Compute

Before you buy anything, diagnose which constraint is actually limiting your experience. The fix depends entirely on the diagnosis.

Symptom you observe	Likely bottleneck	What fixes it	What does not
Model fails to load, or system pages to disk	Capacity	More RAM or VRAM of the right kind, or a smaller model	Faster CPU or GPU
Model loads, but generation crawls with GPU busy	GPU compute or VRAM bandwidth	Faster GPU or more VRAM bandwidth	More system RAM
GPU underused while CPU churns	Offload or CPU-only path	Better placement, more VRAM, or unified memory	More system RAM
Model fits and runs fast, but context or concurrency is limited	KV-cache or total memory headroom	More VRAM, or more system RAM if the model is CPU-resident	Faster GPU alone

The way to observe the difference is to watch memory utilization and token-generation rate in the runtime's own reporting rather than guessing. Run a generation, watch all three, and the bottleneck usually announces itself.

Here is the editorial judgment most buyers need to hear: if you think you need more system RAM for local AI, you probably need more VRAM or a faster memory subsystem instead. More system RAM is the correct purchase only when your model fits in VRAM but the OS, runtime, and concurrent applications are paging. That condition exists, but it is far less common than the desire to run a bigger model.

System-RAM Tiers and What Each Buys You

System-RAM capacity translates into practical capability tiers. These are tied to model class and placement, not universal recommendations, because the right tier depends on where the model lives.

16GB. Enough for a small VRAM-resident model plus the OS and applications. Fine for light experimentation with 7B–8B models on a GPU, but a tight floor once you offload or try anything larger. If your GPU has 8GB of VRAM and you want to run a 70B model with offload, 16GB of system RAM will not hold the offloaded layers plus the OS.

32GB. Comfortable for an 8B-class model with headroom for the OS and concurrent work. A reasonable baseline for most local-LLM users. It also supports modest offload of larger models, though the bandwidth penalty still applies. Best fit: interactive chat and coding assistance on models up to roughly 14B, fully GPU-resident.

64GB and up. Opens the door to larger models with meaningful offload, or to unified-memory systems that run bigger models entirely in one pool. On a discrete-GPU system, 64GB gives a Q4 70B model room to offload without paging—but not interactive speed. On a unified-memory system like the Framework Desktop with 64GB, you can run models that would never fit a consumer GPU. Best fit: batch workloads, experimentation with large models, or running a model alongside real development work.

128GB-class unified-memory systems. The Framework Desktop with Ryzen AI Max comes in 32GB, 64GB, and 128GB configurations, and NVIDIA positions its RTX Spark and DGX Spark systems with up to 128GB of unified memory and stated capacity for models up to 200B. Those vendor-stated model capacities are positioning, not independent performance guarantees. The Framework community's own testing shows that a 128GB system can load very large models, but generation speed depends on the model's active parameter count and the platform's bandwidth. A 109B-parameter model with a small active subset runs far faster than a dense 70B model on the same hardware.

The diminishing-return point matters. Beyond what your model class and context actually need, extra system RAM buys headroom you may never use. A 128GB machine running an 8B model is not faster than a 32GB machine running the same model; the model fits in either case, and the extra capacity sits idle.

Discrete GPU, Unified Memory, or More RAM: The Decision Rule

The decision boundary comes down to what your workload actually hits.

Add system RAM when your model fits in VRAM but the OS, runtime, and concurrent apps are paging. This is the cheapest fix and the one most buyers overestimate. If your 8B model runs fine in VRAM but the system stutters because the browser and editor are competing for 16GB of system RAM, adding another 16GB is the right call. It is also the least exciting purchase you can make, which is why it gets skipped in favor of GPU upgrades that change nothing.

Upgrade GPU memory when the model itself exceeds VRAM and you need responsive, fully accelerated inference. This is the real fix for most discrete-GPU users. A Q4 70B model needs roughly 35GB; no 24GB GPU runs it without offload. If you need that model at interactive speed, the answer is a GPU with more VRAM, not more system RAM. NVIDIA's own guidance positions GeForce RTX systems with 6–32GB of VRAM for smaller model development and RTX PRO configurations with 16–96GB for larger work, which is a reasonable map of the discrete-GPU landscape.

Consider a unified-memory platform when you regularly need models larger than any practical discrete GPU and can accept the bandwidth and software-ecosystem tradeoffs. A 128GB unified-memory system can load models that no consumer GPU can touch. The cost is that unified-memory bandwidth, while high, still trails the fastest discrete GPUs, and software support varies by platform.

The flip conditions matter. If your workload is small models with long context, more system RAM for KV-cache headroom may be the right purchase. The KV cache grows with context length, and a long-context session on a modest model can push memory use well past the static weight footprint. If your workload is large models needing speed, only VRAM or unified memory changes the result. No amount of system RAM makes a CPU-offloaded 70B model feel like a VRAM-resident one.

The evidence supports conditional product examples, not universal performance rankings. The reference material does not provide matched cross-platform benchmarks that would justify a tokens-per-second leaderboard. What it does support is the governing rule: buy the bottleneck your workload actually hits, and verify with a memory monitor before spending.

The Decision Rule

Watch your memory utilization and token-generation rate during a representative session. Then apply this sequence:

1. If the model fails to load or the system pages to disk, add memory of the kind your placement needs—VRAM for a GPU-resident model, system RAM for offload or CPU-only inference.
2. If the model fits in VRAM but the system stutters from concurrent apps, add system RAM. This is the only case where more RAM is the clear fix.
3. If the model spills to system RAM and generation crawls, you need more VRAM or a unified-memory platform, not more system RAM.
4. If the model loads and runs fully in VRAM but generates slowly, you are hitting bandwidth or compute, and no memory purchase fixes that.

The spec sheet will not tell you which constraint binds. Your workload, observed while it runs, will.