How to Check Whether Your GPU Is Actually Accelerating Local AI

You installed a local LLM tool, loaded a model, and started typing. The response comes back slowly—sometimes painfully slowly. Task Manager or Activity Monitor shows your CPU pegged near 100 percent while the GPU sits mostly idle. Or worse, you get an out-of-memory error even though the model appeared to load fine.

The core misconception is simple: a GPU that is detected is not the same as a GPU that is being used. Your operating system can see the graphics card perfectly well while the inference engine silently runs the entire model on the CPU. Before you blame the hardware—or spend money on an upgrade—you need to verify whether the GPU is actually doing the work.

This guide walks through a diagnostic sequence that separates a configuration problem from a genuine VRAM or compute shortfall. Work through the checks in order. Each one eliminates the most likely and cheapest-to-fix cause first.

Set a Baseline Before You Change Anything

Before you touch a single setting, record your starting state. Without a baseline, you cannot tell whether a change helped, hurt, or did nothing.

Write down the following before any test run:

• Hardware and OS: GPU model, VRAM or unified-memory capacity, system RAM, operating system version, and whether you are on a laptop or desktop.
• Software versions: GPU driver version, inference application and version, and the underlying runtime or backend it uses.
• Model details: Exact model name, file format, quantization level, and context length setting.
• Power state: Whether the system is on battery or plugged in, and which power mode is active.
• Test prompt: Use the same prompt for every comparison run.

Run a generation long enough to produce a stable reading—at least a few hundred tokens, not a one-sentence reply. Short bursts hide throttling and warm-up effects. Repeat the run twice to confirm the result is consistent.

Measure two separate numbers:

• Time to first token (TTFT): How long the model takes to start responding after you send the prompt. This reflects prompt processing.
• Sustained decode speed: Tokens per second during ongoing generation. This reflects how the model feels during a real conversation or document task.

These two numbers behave differently and can point to different bottlenecks. A system can produce the first token quickly while generating subsequent tokens slowly, or vice versa. AMD's own Ryzen AI documentation, for example, describes a hybrid arrangement where the NPU handles time-to-first-token and the iGPU handles token generation for LLM inference. Measure both, but treat sustained decode speed as the number that defines your daily experience.

With that baseline recorded, you can start the diagnostic sequence.

Confirm the Driver and Runtime Backend First

Start with the software stack, because it is the most common cause of apparent GPU failure and the cheapest to fix.

Verify the driver is current. Each GPU vendor has its own driver channel, and the version matters more than you might expect. NVIDIA has described meaningful load-time and response-time gains for LM Studio when paired with CUDA 12.8, for example. An outdated driver can mean your runtime silently picks a slower path or fails to engage the GPU at all.

Confirm the inference runtime is installed for your specific GPU vendor. This is where many local-AI setups break. The backend is vendor-specific:

Platform	Typical acceleration path
NVIDIA GeForce RTX / RTX PRO	CUDA
AMD Radeon discrete GPUs	ROCm
AMD Ryzen AI integrated GPUs	Vulkan
Intel Core Ultra / Arc	IPEX-LLM
Apple silicon	Host-process execution of the inference engine

A runtime that accelerates one GPU family may run CPU-only on another. AMD's ROCm documentation explicitly covers different backends for different hardware—Vulkan for integrated GPUs on Ryzen AI systems, ROCm for discrete Radeon cards. Intel's IPEX-LLM is a separate PyTorch library with its own optimization path. If you are using a tool that works on your friend's NVIDIA system but you have an AMD or Intel GPU, the tool may not have selected the right backend for your hardware.

Check what the application actually selected. Most local-LLM tools expose their backend choice somewhere. Look for:

• The application's settings or model-loading screen, which usually shows which backend or acceleration path is active.
• The logs generated when a model loads. Many tools print a line indicating GPU offload is enabled or disabled.
• Any "runtime" or "engine" selector in the tool's configuration.

The application layer matters as much as the underlying engine. Tools like LM Studio, Ollama, and Lemonade sit on top of lower-level runtimes—llama.cpp is a common foundation—and each application may handle backend selection differently. Docker Model Runner, for example, uses an engine built on llama.cpp and achieves GPU acceleration on Apple silicon by running the inference engine as a host process. That is a specific implementation choice that does not transfer automatically to other platforms.

What you should observe: The application reports that GPU acceleration or offload is enabled, and the logs name the backend you expect for your hardware.

What it means if this check fails: You have found a configuration problem. Fix the driver, install the correct runtime, or select the right backend in the application before diagnosing anything else.

Check Where the Model Is Actually Placed

With the backend confirmed, the next question is where the model weights and computation actually live. This is the local LLM CPU vs GPU question, and the answer is often partial.

GPU offloading is the technique of running part of a model on the GPU and part on the CPU when the full model does not fit in VRAM. NVIDIA describes this as a way to run models larger than the GPU's memory by taking advantage of acceleration regardless of model size. It is a legitimate and common configuration—but you need to know which fraction is where.

Read the offload information. Most local-LLM tools report how many model layers are placed on the GPU versus the CPU. This is usually visible in the model-loading screen or logs. If the tool says something like "loaded 20 of 33 layers to GPU," you have partial offload. The model is using the GPU, but a significant portion of the computation is still happening on the CPU.

Compare the model file against your VRAM. The model's file size and quantization level determine how much memory it needs. A model that requires roughly 19GB of VRAM for full GPU placement—a figure NVIDIA cites for one quantized model example—will only partially offload on a 12GB card. The tool may still show GPU activity, but the CPU-resident layers become a bottleneck during generation.

Treat file size as a clue, not proof. Runtime memory use does not equal file size. The model weights are only part of what occupies VRAM during inference. The context window, its key-value cache, batching overhead, and backend memory allocation all add to the total. A model that fits by file size can still fail at a longer context, while a model slightly larger than VRAM may be perfectly usable if you accept slower generation. What matters is not whether the model fits perfectly, but whether the measured decode speed meets your needs.

Distinguish two different problems:

• Model too large for VRAM: The model partially offloads, GPU activity is visible, but generation is slow because CPU layers bottleneck the process.
• Model misconfigured to run on CPU: The model fits in VRAM, but the tool never placed it there. This is a settings or backend problem, not a hardware limit.

These require different fixes. The first might mean a smaller or more quantized model. The second means fixing the configuration.

What you should observe: The tool reports the number of layers or percentage of the model placed on the GPU, and that number is consistent with your VRAM capacity.

What it means: If the model is fully on the GPU and still slow, move to the next check. If it is partially offloaded, you have found a VRAM-fit problem. If it is entirely on the CPU despite fitting in VRAM, you have a configuration problem.

Read VRAM and Utilization Signals Correctly

Monitoring tools can mislead you if you do not know what you are looking at. Here is what to check and how to interpret it.

VRAM committed by the model. Open your vendor's monitoring tool—NVIDIA's control panel or task manager GPU section, AMD's Adrenalin software, or the OS-level GPU monitor. During inference, you should see a significant chunk of VRAM committed by the model. If VRAM usage is near zero while the model runs, the model is not on the GPU.

GPU utilization during generation. This is where readings get tricky. A GPU that spikes to high utilization during prompt processing but drops toward idle during token generation is a sign that the model is largely CPU-resident. The GPU handles the initial burst of work, then waits while the CPU grinds through generation. Conversely, sustained GPU utilization during token generation indicates the model is actively being accelerated.

CPU behavior. If the CPU is pegged near 100 percent while the GPU idles, that is a strong signal that inference is CPU-bound. But do not treat it as proof by itself. CPU usage can remain high even with GPU offload, because the CPU still handles prompt processing, sampling, and the layers that remain in system memory.

The misleading number to watch for: High GPU utilization alone is not proof of good acceleration. A GPU can show high utilization while the overall process is bottlenecked by CPU-resident layers, memory bandwidth limits, or inefficient batching. The outcome that matters is decode throughput—tokens generated per second—not the utilization percentage.

VRAM versus system memory. Out-of-memory errors can come from either the GPU buffer or the host RAM used for offload. If a model partially offloads, the CPU-resident layers consume system memory. A model that "loads" but then errors during generation may have exhausted system RAM, not VRAM. Check both when you hit memory errors.

What you should observe: VRAM usage roughly matches the model size plus context overhead during inference, GPU utilization is sustained during generation rather than spiking only at the start, and the CPU is not pegged while the GPU idles.

What it means: These signals corroborate the backend and offload logs from the previous checks. If the logs say the GPU is engaged and the memory behavior matches, you have confirmed acceleration. If the signals contradict the logs, trust the contradiction as a sign that something is misconfigured—and recheck the backend selection.

Measure Token Generation Against Your Own Baseline

The decisive observable for local LLM acceleration is tokens per second during sustained generation, compared against your baseline. Load percentages and utilization readings are proxies; token throughput is the actual outcome.

Run a controlled comparison. The most useful test isolates whether a change is software or hardware. Run the same model with GPU acceleration explicitly enabled, then again with it disabled or forced to CPU. Keep the prompt, context length, and quantization identical. The difference tells you what your GPU is contributing.

Interpret the gap, not the absolute number. Exact token rates depend heavily on model size, quantization, hardware class, context length, and power state. A fixed threshold like "8B models should hit X tokens per second" is unreliable across that much variation. What matters is the delta between your CPU-only baseline and your GPU-enabled run, and whether the GPU-enabled result meets the speed you actually need.

If the GPU-enabled run is barely faster than CPU-only, you have a bottleneck elsewhere—possibly partial offload, memory bandwidth, or thermal limits. If the GPU-enabled run is several times faster but still too slow for your use case, you have a genuine capacity problem.

Watch for low utilization during decode. A correctly accelerated setup can still show modest GPU utilization during token generation. Decode is often memory-bandwidth-bound rather than compute-bound, especially with smaller models or long contexts. The GPU may be working hard without showing 100 percent utilization. Do not interpret moderate utilization as proof that the GPU is idle—check the backend logs and the token rate instead.

What you should observe: A tokens-per-second reading that is clearly faster than your CPU-only baseline and consistent across repeated runs.

What it means: If token generation is slow despite confirmed GPU placement and healthy VRAM usage, the problem is likely thermal or power related—the next check.

Rule Out Thermal and Power Limits

A correctly configured GPU that is fully engaged can still run slowly when power limits or thermals cap sustained clocks. This is especially common on laptops and compact systems, where the cooling and power envelope is much smaller than a desktop card's.

Why short tests miss it. A GPU can boost to high clocks for a short burst, then drop significantly once temperatures rise or the power limit is reached. A benchmark that runs for 30 seconds may show excellent performance while a 10-minute generation reveals severe throttling. Long generations are precisely the workload that exposes this.

Check clock behavior during a sustained run. Use your vendor's monitoring tool to watch GPU clock speed and temperature over several minutes of continuous generation. Look for:

• Clock speeds that start high, then drop and stay dropped
• Temperatures that climb to the thermal ceiling and hold there
• Power draw that hits the board's power limit

Laptop versus desktop. Laptop GPUs operate within strict power and thermal envelopes. A laptop GPU may sustain lower clocks than its desktop counterpart with the same name, and that is by design—the cooling system simply cannot dissipate the heat. If you are running local AI on a laptop, expect sustained performance to be lower than a short benchmark suggests.

Distinguish throttling from a compute shortfall. These are different problems with different fixes:

• Thermal or power throttling: The GPU is capable of more, but the system limits it. The fix might be better cooling, a power-setting change, a laptop cooling pad, or accepting that the system class cannot sustain the workload.
• Genuine compute or VRAM shortfall: The GPU is running at its limits and still is not fast enough. The fix is different hardware.

What you should observe: Clock speeds and temperatures during a sustained run of several minutes, not a short burst.

What it means: If clocks drop and temperatures hit the ceiling, you have a thermal or power configuration problem. If clocks hold steady at reasonable temperatures and generation is still slow, the hardware itself is the limit.

Decide: Reconfigure, Shrink the Model, or Upgrade

By this point, you should know which of three situations you are in. Each maps to a different fix.

Situation 1: The GPU was never engaged. The backend was wrong, the driver was outdated, or the model was placed on the CPU despite fitting in VRAM. Fix the software configuration. This is the cheapest fix and the most common cause of apparent GPU failure. Reinstall or update the driver, install the correct runtime for your GPU vendor, and verify the application reports GPU offload enabled.

Situation 2: The model does not fit in VRAM. Partial offload is confirmed, and CPU-resident layers are bottlenecking generation. Before buying hardware, try a smaller model or a more aggressively quantized version of the same model. Quantization reduces the model's memory footprint at some cost to output quality—the tradeoff is usually acceptable for experimentation and many practical tasks. NVIDIA's own guidance frames this as a size-versus-quality-versus-performance tradeoff: larger models generally produce higher-quality responses but run more slowly.

The decision is not simply "full offload is always faster." A smaller quantized model that fits entirely in VRAM will often generate tokens faster than a larger model that partially offloads to the CPU—but not always. The larger model may produce better responses, and if you are willing to wait, partial offload may be the right tradeoff. Run the comparison yourself with your actual prompt and context length before deciding.

Situation 3: Acceleration is confirmed but still too slow. The model is on the GPU, VRAM usage is healthy, and token generation is clearly faster than CPU-only—but it is not fast enough for your needs. Check thermals and power limits first, especially on a laptop. If throttling is the cause, the fix may be a setting change, better cooling, or a different system class rather than a bigger GPU.

Only when the GPU is running at its sustained limits and still cannot deliver the speed you need does a hardware upgrade become the rational move. Define that speed target before you shop. "Faster" is not a requirement; "at least 20 tokens per second for an 8B model with a 4K context" is. If you cannot name the number you need, you cannot know whether any purchase will satisfy it.

The decision rule, compactly:

If you observe	The problem is	The fix is
GPU idle, CPU pegged, backend missing or wrong	Configuration	Fix driver, runtime, or backend settings
Partial offload, model larger than VRAM	VRAM capacity	Smaller or more quantized model, or accept slower generation
GPU engaged, VRAM healthy, clocks dropping, high temps	Thermal or power limit	Cooling, power settings, or different system class
GPU engaged, VRAM healthy, clocks steady, still below your target	Genuine compute shortfall	Hardware upgrade

Most apparent GPU failures are configuration problems, not hardware shortfalls. Work through the software fixes before you spend money. A GPU that was never engaged will not be fixed by a bigger GPU—the new card will sit just as idle if the backend still is not configured. And a model that does not fit in VRAM may run perfectly well in a smaller quantization, saving you the cost of an upgrade entirely.

Only when you have confirmed the GPU is engaged, the model fits, thermals are healthy, and the hardware is running at its sustained limits should you conclude that the workload genuinely exceeds what your current system can deliver. That is the point where a hardware purchase becomes justified—not before.