← Deep dives

Deep dives · 2026-11-14 · Advanced · By bombong

MoE expert offloading: running huge models beyond the GPU

MoE models use only some experts per token, but the weights of all experts still have to be stored somewhere. Expert offloading keeps experts in CPU memory or on SSD and moves them to the GPU only when needed, so huge models can run with limited VRAM. The key is how to hide the wait for PCIe transfers.

Contents
Key points
  • MoE saves compute but not memory. Expert offloading exploits this gap by keeping unused experts outside the GPU.
  • The bottleneck is PCIe transfer. Which experts are needed is known only after the router computes it, so the transfer sits on the critical path of computation.
  • Solutions fall into three levers: what to keep on the GPU (residency and caching), when to fetch ahead (prediction and prefetching), and where to compute (GPU–CPU split).
  • Prefetching is not always beneficial. Research suggests that with a GPU cache, predictions can become less accurate and waste bandwidth, so request scheduling has emerged as an alternative.
  • The best policy changes with model sparsity, CPU and GPU performance, VRAM and bandwidth. Benchmark numbers from someone else's setup should not be carried over directly to your own environment.

Why it was needed: compute went down, memory did not

MoE (mixture of experts) is an architecture that places several small neural networks, called 'experts', inside a model, and for each token a router (a small selector that decides which experts to use) picks only a few of them to compute. Since the active parameters actually used to process one token are only a fraction of the total, a larger model can be built for the same amount of compute. The problem is that it is not known in advance which experts will be chosen. So all experts, including ones that may not be used, have to be loaded into memory.

As a result, MoE models easily exceed the memory of a single accelerator. Specter paper starts from the problem that MoE models commonly exceed the memory capacity of a single accelerator. AdaOffload paper points out that MoE inference is bound by GPU memory, and that serving ever-larger frontier models requires more GPUs even though computation is sparse. On consumer GPUs or edge devices, the situation is even tighter. SMoE paper and OD-MoE paper directly target edge and low-cost hardware with small device memory.

Expert offloading exploits this gap. It keeps expert weights in CPU-side memory (host DRAM) and caches only a frequently used subset in GPU memory (VRAM). If a needed expert is not on the GPU, it is fetched at that point. OD-MoE paper summarizes existing offloading this way: expert parameters are stored in CPU memory, and a popular subset is cached in GPU memory. The cost is also clear. According to Specter paper, moving experts down to host memory creates a severe PCIe (the data path connecting CPU and GPU) I/O bottleneck. Because routing is determined by the data, transfers remain directly on the critical path of computation.

Core idea: a small workbench and a large warehouse

A kitchen analogy makes it easier to understand. The GPU is the small workbench right in front of the cook, CPU memory is a large storeroom at the end of the hallway, and the SSD is a basement storeroom even farther away. Cooking (token computation) is fastest on the workbench, but the workbench holds only a few ingredients. When an order (token) comes in, the order slip (router) tells which ingredients (experts) are needed, and if they are not on the workbench, someone has to go to the storeroom and back.

In this analogy, there are three broad ways to improve. First, keep frequently used ingredients on the workbench (residency and caching). Second, guess the next order and bring ingredients in advance (prediction and prefetching). Third, instead of moving them to the workbench, process them right away at a small counter next to the storeroom, that is, on the CPU (CPU compute offload). Most offloading research is about how to combine these three levers.

The analogy also shows what this design trades off. Instead of enlarging the workbench (buying more GPUs), it accepts round-trip time. So the goal is not to eliminate round trips but to keep the cook from sitting idle during them. Overlapping transfer and computation to hide wait time is the common strategy of almost every system.

How it works: what happens in one layer

Here is the step-by-step flow through a single MoE layer. First, the router picks the experts to use for each token. If a chosen expert is in the GPU cache (cache hit), it is computed on the GPU right away. If not (cache miss), one of two things happens: the weights are moved from CPU memory to the GPU over PCIe and then computed, or the expert is computed on the CPU without moving the weights. In configurations that keep the expert pool itself on SSD, there is an extra step of reading from SSD into DRAM. PagedMoE paper addresses this kind of SSD-resident configuration.

'Doing things in advance' is layered on top. If the experts to be used in the next layers are predicted and moved ahead of time while the current layer is being computed, the transfer time is hidden behind computation. OLED-MoE paper notes that existing MoE serving systems rely on this approach of predicting and loading the next layer's experts while computing one layer. When predictions are right, the wait disappears; when they are wrong, unnecessary transfers eat up bandwidth and GPU memory.

COPE paper splits this design space into two independent axes. One is expert residency (which experts to place where), and the other is the MoE execution strategy (where and how to compute). The diagram below draws these two axes and the prediction path for a single layer. In practice, when looking at an offloading engine, breaking it down into three questions—'what to keep, when to fetch, where to compute'—makes the structure clear.

Performance is usually measured with two metrics. TPOT (time per output token, the time to generate one output token) is the speed a single user experiences, and throughput (tokens/second) is the total output of the system when many requests are batched. Which design is good depends on which metric is targeted.

Offloading flow in a single MoE layerToken → Router; Router → GPU cache(Hit); GPU cache → GPU compute; CPU memory → GPU cache(PCIe transfer); CPU memory → CPU compute(CPU execution); SSD → CPU memory(Read); Router → Prediction · prefetch; Prediction · prefetch → CPU memory(Request ahead); GPU compute → To next layer; CPU compute → To next layerTokenRouterExpertselectionGPU cacheVRAM ·frequently usedexpertsGPU computePrediction ·prefetchNext-layerexpertsCPU memoryHost DRAMCPU computeProcess withoutmovingSSDExpert pool(optional)To next layerHitPCIe transferCPU executionReadRequest ahead
Offloading flow in a single MoE layer If the expert chosen by the router is in the GPU cache, it is computed immediately; if not, it is moved from CPU memory or computed on the CPU. The dashed line is an optional flow that predicts and fetches the next layer in advance.

How it evolved: prediction, substitution, scheduling, expansion

The foundational papers are all from 2025–2026, so this section summarizes what was new among them. The first line of work is 'predicting further ahead and more accurately'. SP-MoE paper states that it is the first to co-design speculative decoding (where a small draft model writes several tokens first and a large model verifies them all at once) with offloading. Verifying several tokens at once increases the number of experts needed and intensifies competition for CPU-GPU bandwidth, so it uses the structural correspondence between the draft model and the original model to prefetch experts before verification. Adding a policy that limits per-layer prefetch depth and an asynchronous pipeline, it reduced TPOT by 1.07–3.5x compared with existing methods. Specter paper runs a lightweight quantized draft model on the GPU to look ahead at the experts that will be used over several future steps. It reported an average 1.50x (up to 1.88x) TPOT improvement for single requests. OD-MoE paper pushed prediction to the limit. It splits expert loading and computation across multiple edge nodes, predicts expert activations several layers ahead with 99.94% accuracy, loads them just before they are needed, and unloads them right after use. With a design that has no cache at all, it reached about 75% of the decoding speed of a setup that caches everything on the GPU, on a 10-node testbed.

The second line of work asks, 'does it have to be that exact expert?' SMoE paper does not treat offloading purely as a scheduling problem and brings expert importance into the decision. When a low-importance expert is selected but is not on the GPU, it computes with a functionally similar expert already in the cache instead. Rather than reducing transfers themselves, it approximates the computation. It reported a 48% reduction in decoding latency, a cache hit rate above 60%, and nearly lossless accuracy.

The third line of work is a counterargument to prediction. Bevy paper argued that prefetching can actually hurt when there is an expert cache on the GPU. The more needed experts the cache holds, the less accurate the predictions that must cover what is outside the cache become. So it chose request scheduling instead of prediction. Among concurrent requests, letting those whose experts are ready run ahead to later layers (runahead) causes those requests to reveal the experts they need next 'actually, not by prediction'. Meanwhile, it computes the ready requests while fetching the missing experts. It raised decode throughput by up to 3.46x over llama.cpp.

The fourth line of work extends the scope of application. OLED-MoE paper points out that in semi-autoregressive diffusion LLMs (language models that work like diffusion models), which remove noise block by block, the set of experts used within one iteration grows, making it hard for per-layer prefetching to finish in time. Instead, it exploits the large overlap in expert routing between adjacent iterations, using token confidence to choose experts that will be reused and keep them on the GPU. Unavoidable misses are covered by CPU-GPU cooperative execution. AdaOffload paper targets the problem that existing offloading systems do not scale in performance as GPUs are added. It co-designs GPU-CPU parallelism and cooperative execution for both decode and prefill (the stage that processes the input prompt all at once), and adapts by observing the workload at runtime. On five platforms including L4, Pro6000 and B200, and open-weight models from 284B to 753B, it raised throughput by 1.9–15.5x.

Where it is heading: agents, SSDs, policy selection

Three trends stand out in recent research. The first is local agents. Ostinato paper addresses running agents on workstations with a few consumer GPUs. An agent does a short prefill every time it receives a tool result and then decodes again, but existing CPU-GPU engines are fast at only one of the two. For example, llama.cpp's expert cache skips the cache for forward passes longer than 32 tokens in order to protect experts for decode, so each tool output means a 2.1–2.8 second wait. Switching to the default layer placement shortens the wait but makes decode 1.5–1.6x slower. Ostinato found that although short prefills touch many experts, 84% of routing goes to a stable set of experts that fits on the GPU, and it computes this set on the GPU on every forward pass. On four RTX 4090D cards, compared with llama.cpp and KTransformers, it cut the median wait after tool outputs by 22–55% and finished sessions 1.4–2.5x faster.

The second trend pushes the hierarchy down one more level. PagedMoE paper places the routed expert pool on SSD and uses DRAM as a cache. It reduces the problem of SSD I/O remaining on the critical path by varying the precision (number of bits) of experts per token. It stores the low-precision version of an expert as the leading part (prefix) of the high-precision version, so bits already in the cache are reused and only the missing bits are read. At an average of 2.5 bits, it achieved quality close to GPTQ-INT4 with Qwen3.5-397B-A17B, and recorded 8.89 tokens per second with a 24GiB expert cache. This shows quantization and offloading merging into a single cache design rather than being separate decisions about 'how much to shrink' and 'where to put it'.

The third trend is automating policy selection. COPE paper points out that policy rankings change as model sparsity, CPU and GPU throughput, VRAM capacity and host-device bandwidth change. To avoid the cost of implementing every candidate, it measures combinations of operations on the target platform and combines them with offline routing traces to estimate performance. Compared with the evaluated baselines, it improved the accuracy of picking the fastest policy by 24–100%. This signals a shift from searching for one right policy to searching for a way to choose a policy for each environment.

How many experts stay on the GPU

← Load only when used, unload immediatelyEverything resident on GPU →
1234
  1. OD-MoE: no cache
  2. SMoE · Bevy: cache popular experts
  3. Ostinato: stable set kept resident
  4. Full GPU deployment
Further left saves more GPU memory and relies more on transfers; further right keeps more resident on the GPU. Positions are only a relative comparison, not measurements.

Limitations and open issues

First, the numbers are tied to their environments. The speedup figures above come from different baselines (llama.cpp, the previous best system, a full GPU-cache setup), different metrics (TPOT, throughput, session completion time) and different hardware. As COPE paper shows, the policy ranking itself can flip when hardware and models change. Papers should not be ranked by placing their numbers side by side.

Second, prediction is not free. Wrong predictions waste bandwidth and GPU memory, and Bevy paper argued that prefetching can actually hurt when a cache is present. OLED-MoE paper also noted that in diffusion LLMs, existing prefetching effectively degrades into on-demand loading, increasing decoding latency. On the other hand, OD-MoE paper managed without a cache thanks to a highly accurate predictor. Whether prediction pays off depends on the shape of the workload.

Third, optimizations that change the computation must be distinguished from those that do not. SMoE paper's expert substitution and PagedMoE paper's lower precision change the values the model actually computes. The reported quality is 'nearly lossless' or 'close', not identical. Ostinato paper states that it does not change the model's computation. For tasks where accuracy verification is demanding, this difference decides the choice.

Fourth, single-request and multi-request settings are different problems. Specter paper delivered results for single-request inference, while Bevy paper only works when there are concurrent requests. The right design depends on whether it is a local LLM used by one person or a server handling many users.

What to consider when choosing in practice

It is clear when expert offloading makes sense: when the model weights are larger than the VRAM you have, and accepting some latency is better than buying more GPUs. Typical cases are running a large open-weight MoE on a local workstation, an edge device or a small number of GPUs. Conversely, if the model already fits entirely in VRAM and latency matters, offloading is a loss. Even OD-MoE paper ran at about 75% of the speed of a deployment with everything cached on the GPU.

When choosing, first look at the shape of your workload. For single-user chat, use TPOT as the criterion; if you batch multiple requests, use throughput. When short prefills and decodes alternate, as with agents, choosing an engine by decode speed alone misses the wait that occurs on every tool call, as Ostinato paper points out. If you plan to use speculative decoding as well, you also need to consider the bandwidth contention pointed out by SP-MoE paper.

There are three common mistakes: trusting benchmarks from different hardware as-is, turning on approximation techniques that change the model's computation without validating quality, and not accounting for CPU-side resources (memory capacity, core count, NUMA configuration). AdaOffload paper included CPU performance and NUMA configuration as evaluation variables, and research on distributed setups such as vExpert separately addresses DRAM usage and PCIe sharing. It helps to go through the checklist below before deployment.

Checklist before adopting offloading

Need

  • Does the model really not fit in VRAM?
  • Is accepting latency better than adding GPUs?

Workload

  • Single requests or concurrent requests?
  • Is the criterion TPOT or throughput?
  • Do prefill and decode alternate, as with agents?
  • Is speculative decoding also used?

Validation

  • Have you confirmed approximation techniques don't change quality?
  • Have you measured the policy on your own hardware?
  • Have you accounted for CPU memory, cores and NUMA configuration?
The issues covered in the main text, grouped into items to check before deployment.

Further reading

If you are new to this, start with the problem setup section of OD-MoE paper. It briefly lays out the basic structure, 'keep experts in CPU memory and cache only popular experts on the GPU', along with its limits. Reading SMoE paper next gives you the perspective that 'there is also the option of slightly changing the computation, not just scheduling.'

To go deeper into the prediction line of work, read SP-MoE paper and Specter paper together. Both use the draft model from speculative decoding for expert prediction, but one emphasizes bandwidth contention during the verification step, while the other emphasizes looking far ahead with a quantized draft. Reading Bevy paper after that gives a balanced view that includes the counterargument to prediction.

If you are interested in server scale or production deployment, read AdaOffload paper and COPE paper; if you run local agents, read Ostinato paper; and if memory is so tight that you need to use SSDs, read PagedMoE paper. If you work with models that generate differently, such as diffusion LLMs, OLED-MoE paper shows why existing assumptions break down.

Related terms

MoE Active parameters VRAM Speculative decoding Quantization Local LLM Inference engine Throughput (tokens/s)

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.