How to Fit a Local LLM in Limited GPU Memory

You downloaded a model, loaded it into your local LLM runtime, and got one of three results: an out-of-memory error, a model that loads but crawls, or a model that technically runs while your system grinds through swap. None of those outcomes tells you whether your GPU is genuinely undersized or whether the model and runtime settings are wasting the memory you already have.

The answer matters because it determines whether the fix is a configuration change or a hardware purchase. Before you start shopping for more VRAM, run a diagnostic sequence: measure what memory you actually have, apply the highest-leverage settings changes first, and observe the tradeoff between model quality, speed, and fit. In most cases, you can fit a local LLM in VRAM by adjusting quantization, context length, and GPU offload settings. In some cases, you genuinely cannot, and the measurement tells you that too.

Why a Model That "Fits" Still Runs Poorly

The first mistake is treating model fit like a file-size check. A model's quantized weights are only part of the memory bill. During inference, the runtime also needs space for temporary buffers, compute graphs, and the KV cache, which stores the attention state for the conversation so far. The KV cache grows with context length, which is why a model that loads cleanly with a short prompt can fail or slow dramatically once you feed it a long document.

That means "fitting" is a runtime condition, not a static property of the model file. A model that barely loads into VRAM may still be impractical at the context length or generation speed you actually want. When the model spills part of its weights or cache into system memory, every token generation that touches those spilled layers has to cross the memory bus, and your tokens-per-second number collapses.

Before changing anything, identify which failure state you are in:

• Won't load: The runtime reports an out-of-memory error before or during model loading.
• Loads but spills: The model loads, but part of it runs from system RAM. Generation is slow, and you may see system memory usage climb.
• Loads but runs too slowly: Everything appears resident in VRAM, but token generation is slower than the workload can tolerate.

Each state has a different fix. The diagnostic sequence below handles all three, but you need to know which one you are starting from.

Measure What Memory You Actually Have

The advertised VRAM number on a GPU box is not the same as the memory available to your model. The desktop environment, other running applications, and the driver itself reserve a portion of VRAM before your LLM runtime even starts. On a gaming laptop, the integrated GPU may claim a chunk of system memory for display output. On unified-memory systems, the situation is more complicated: usable graphics memory can be limited by firmware settings and runtime behavior rather than by total RAM alone.

Start by checking what your system actually reports. Your GPU driver's control panel or a system monitoring tool will show total VRAM, current usage, and what is reserved. On Windows, the Task Manager's GPU section shows dedicated GPU memory usage. On Linux, tools like nvidia-smi for NVIDIA GPUs or rocm-smi for AMD GPUs report memory usage directly. For unified-memory systems, check the runtime's own memory reporting, since the operating system's memory counters may not distinguish what the GPU can actually address.

A practical shortcut is to use a diagnostic tool that measures real local-LLM fit against your GPU, VRAM, RAM, and CPU rather than guessing from model file size. Tools like VRAM Check CLI on Windows run an actual inference workload and report which models fit your hardware. VRAMFit: LLM Calculator serves a similar purpose as an estimation aid, including for Apple Silicon systems. These tools are useful for establishing a baseline, but treat their results as estimates: they cannot predict every runtime's overhead or your specific context-length needs.

Before you change any settings, record a baseline:

• Model name and quantization level
• Context length setting
• Number of layers offloaded to the GPU (if your runtime exposes this)
• Observed tokens per second during generation
• Whether the model stays fully resident in VRAM or spills to system memory

This baseline is your reference point. Without it, you cannot tell whether a setting change actually helped.

Cut Memory With Quantization First

Quantization is the highest-leverage, lowest-effort adjustment you can make. It works by reducing the number of bits used to store each weight in the model. A model stored at 16-bit precision uses two bytes per weight; the same model at 4-bit precision uses half a byte per weight. Since weights dominate the memory bill, quantization directly shrinks the largest consumer.

The tradeoff is response quality. Lower bit widths mean less precision, which can degrade the model's output, especially for complex reasoning, math, or tasks that require exact recall. The quality loss is not uniform across models or tasks, and many users find that modern 4-bit quantization is surprisingly close to the full-precision model for everyday conversation and drafting. But the loss is real, and at very low bit widths it becomes obvious.

The practical ladder is simple: try the next-lower quantization of the same model before switching models or buying hardware. If a 7-billion-parameter model fails to load at 8-bit, try the 4-bit version of the same model. If that fits but runs slowly, look at context length next. Only after you have exhausted the quantization ladder for the model you actually want should you consider whether a different model or different hardware is the answer.

One warning: vendor model-size examples assume a specific quantization and context length. When a GPU vendor says a card can run a particular model, that claim is tied to a specific configuration. The same model at a higher quantization or with a longer context setting will need more memory. Treat vendor examples as starting points, not guarantees.

Trim Context Length and Runtime Overhead

If quantization alone does not free enough memory, the next adjustment is context length. The KV cache grows roughly linearly with the number of tokens the model can attend to. A model configured for a 32,000-token context reserves far more memory than the same model at 4,000 tokens, even if your typical conversation only uses a few hundred.

The fix is straightforward: lower the context length setting in your runtime to something that still serves your use case. If you are having a back-and-forth chat, a few thousand tokens of context is often plenty. If you are feeding the model long documents, you need more, but you may not need the maximum the model supports.

Context length also interacts with how runtimes allocate memory. Some runtimes reserve buffers up front based on the configured context, which means a model that fails to load at a high context setting may load cleanly at a lower one. The same applies to batch size or parallel-processing values where your runtime exposes them. Lowering these settings reduces the memory the runtime reserves before generation starts.

This is a durable setting change, not a one-off workaround, when the shorter context still serves your actual workload. If you routinely need long-context analysis, trimming context to fit the model is the wrong trade, and you should look at the hardware question instead. But if you are running interactive chat or code completion, a shorter context is often invisible in daily use.

Offload Layers to System Memory When Needed

If quantization and context trimming still leave the model too large for VRAM, GPU offloading is the next option. Offloading runs part of the model on the GPU and part on the CPU, allowing a larger model to load when it would not otherwise fit. The runtime keeps some layers resident in VRAM and spills the rest to system memory, where the CPU processes them.

The tradeoff is speed, and it is severe. Layers running on the CPU and system RAM generate tokens far more slowly than GPU-resident layers. The more layers you offload, the slower generation becomes. A model that is mostly on the GPU with a few layers spilled may be noticeably slower but still usable. A model that is mostly on the CPU will feel like a different product entirely.

Most runtimes that support offloading expose a setting for how many layers to keep on the GPU. In llama.cpp-based runtimes, this is often the --n-gpu-layers or -ngl flag, and in LM Studio it appears as a GPU offload slider. The mechanism is the same across runtimes: you are choosing how much of the model runs on fast GPU memory versus slow system memory.

Offloading is worth it when you need a larger model for occasional, non-interactive work: batch processing, long-running background generation, or tasks where you can wait minutes instead of seconds. It is not worth it for interactive chat, where latency matters and a model that takes thirty seconds per response will frustrate you into abandoning it. If you are offloading more than half the model layers and still need interactive speed, the GPU is the bottleneck.

Read the Result: Is It a Config Problem or a Hardware Limit?

After applying quantization, context, and offload changes, re-measure the same metrics you recorded in your baseline: tokens per second, whether the model stays resident in VRAM, and whether the output quality still serves your purpose.

The interpretation is straightforward:

• If the model now runs at usable speed and acceptable quality, the constraint was configuration, not hardware. You have successfully fit a local LLM in your available VRAM by adjusting settings. Keep the configuration that works and move on.
• If the model only runs after heavy quantization or offload that degrades quality or speed below what your workload needs, the GPU is the bottleneck. No setting change will fix this, because the problem is the total memory available, not how you are using it.

The observable signals that point to a hardware limit are specific: unusable speed even at the lowest acceptable quantization, or quality loss that defeats the purpose of running the model in the first place. If you are running a heavily quantized model that produces noticeably worse output and still generates slowly, you have hit the memory ceiling.

When Reconfiguration Isn't Enough: Upgrade Paths

If the diagnostic sequence confirms a genuine hardware limit, the upgrade decision comes down to buying the bottleneck. For local LLM work, that means one of three directions:

• A discrete GPU with more VRAM. This is the most direct path for NVIDIA and AMD GPU users. A card with 24GB or 32GB of VRAM opens up substantially larger models than a 12GB or 16GB card. As a reference point, NVIDIA's GeForce RTX 50-series lineup spans from 16GB on the RTX 5080 to 32GB on the RTX 5090, and vendor material positions these cards for local AI workloads. The RTX PRO workstation line extends further for development work.
• A unified-memory system with more addressable memory. Systems built around AMD's Ryzen AI Max processors, such as the Framework Desktop, come in 32GB, 64GB, and 128GB configurations. AMD's Variable Graphics Memory feature can convert system RAM into graphics memory, with the top configuration offering up to 96GB of usable graphics memory. NVIDIA's DGX Spark occupies a similar category with up to 128GB of unified memory. These systems trade raw GPU compute speed for much larger memory capacity, and usable capacity depends on firmware settings and runtime behavior.
• A second GPU. Adding a second card can double available VRAM, but only if your runtime supports spreading the model across cards. Recent llama.cpp versions support tensor parallel for two identical NVIDIA GPUs, which both shares memory and uses both cards for compute. The catch is that the GPUs must be homogeneous, and not all runtimes support multi-GPU inference. A second GPU is a viable path for NVIDIA users running llama.cpp-based tools, but verify your runtime's support before buying.

One caution about vendor capacity claims: when a vendor says a system can run a model of a certain size, that claim assumes a specific quantization and context length. The same hardware will not run the same model at higher precision or with a much longer context. Use vendor examples to understand the category, not to predict your exact workload.

A Decision Rule for Your Setup

The full diagnostic sequence compresses into a repeatable rule:

1. Measure real available memory and record your baseline model, quantization, context, and tokens per second.
2. Quantize to the next-lower bit width of the same model.
3. Trim context length and runtime buffer settings to what your workload actually needs.
4. Offload layers to system memory only if the model still does not fit, and only if you can tolerate the speed loss.
5. Re-measure tokens per second and assess whether output quality still serves your purpose.

Then apply the flip condition: keep your current setup if the adjusted model runs at usable speed and quality. Upgrade only if your workload demands a model that no reasonable setting makes usable.

Do not buy more VRAM before confirming the constraint is memory rather than a runtime or configuration issue. The measurement sequence costs an hour. A GPU purchase costs hundreds or thousands of dollars, and it will not fix a model that was running slowly because you had the context length set to the maximum or the wrong quantization loaded.

Run the measurement and adjustment pass once, on your actual workload, with your actual context needs. The result will tell you whether you need a new configuration or a new GPU, and it will tell you which one before you spend any money.