Mixture of experts (MoE): how to run huge models cheaply

Why it was needed: making a model bigger also makes computation bigger

In deep learning, given enough data, increasing a model's capacity, that is, its parameter count, improves predictions. The problem is that a typical model computes all of its parameters for every input. The Sparsely-Gated MoE paper paper pointed out that in such architectures, when model size and training data grow together, training cost rises roughly quadratically, and the growth of compute resources cannot keep up.

A long-discussed solution is conditional computation. If only part of the network is turned on for each input and the rest is left off, capacity can grow while computation grows less. But the same paper noted that this idea had never actually worked well and summarized why. GPUs are far better at arithmetic than at branching; turning on only some parts shrinks the batch each part receives, which lowers efficiency; network bandwidth between multiple GPUs becomes a bottleneck; and spreading work evenly requires a separate loss term.

MoE is the method that solved these problems through engineering and actually achieved 'large parameter count, small per-token computation.' According to this paper, the mixture-of-experts idea itself has been studied since around 1991, but this 2017 work is the starting point for its place as a component of modern large language models.

Core idea: a general hospital that calls in only the departments needed

Think of a general hospital. It has dozens of departments, but each patient is seen only by the one or two departments that the front desk assigns after hearing the symptoms. The hospital's overall expertise (capacity) is large, but the treatment time spent on any one patient (computation) is small. In MoE, the departments are the 'experts' and the front desk is the 'router' or 'gating network.'

In transformer terms, each layer has a feed-forward network (FFN, a small neural network that transforms each token separately) after attention. Switch Transformer paper replaced this single FFN with multiple FFN experts and had a router decide which expert each token goes to. This separates the total parameters contained in the model from the active parameters actually used in computation when processing one token.

The important point is that the router is not a hand-written rule but a small neural network trained along with the rest of the model. Which expert handles what is not decided in advance. The Sparsely-Gated MoE paper paper observed that after training, experts tend to become substantially specialized by syntax and semantics.

How it works: scoring, selecting, mixing

The way one MoE layer processes a token can be viewed as four steps. First, the token's Representation (the vector computed up to that point) goes into the router. Second, the router produces a score for each expert with a simple matrix multiplication and turns the scores into probability-like values with softmax. Third, only the k experts with the highest scores are selected (top-k). Fourth, the selected experts each compute on the token, and their outputs are combined as a weighted sum using the router scores and passed to the next layer. Switch Transformer paper formalized this computation in equations; the key point is that unselected experts are not computed at all.

This selection happens separately for each token and each layer. Taking Mixtral paper as an example, each layer has 8 FFN experts and the router picks 2 per token. Within the same sentence, different words, and even the same word in different layers, can use different combinations of experts. As a result, a single token can access 47B parameters, but the active parameters actually used during inference are 13B.

When training on multiple GPUs, experts are split across devices. Switch Transformer paper explains that because the weights of different experts are placed on different devices, model weights grow as devices are added, while the memory and computational load on each device stays manageable. In exchange, tokens must be sent to the device holding their expert and the results sent back, which creates communication. This communication cost is one of the key constraints in MoE design.

Development: from MoE between LSTMs to open MoE models

Sparsely-Gated MoE paper is the starting point. It built an MoE layer from up to thousands of feed-forward experts and a trainable gating network, inserted it between stacked LSTM layers, and created models of up to 137B parameters. The authors reported increasing model capacity by more than 1000x with only minor losses in computational efficiency on modern GPU clusters, and achieving better results than the state of the art at the time in language modeling and machine translation with less computation.

Switch Transformer paper brought this idea to the transformer and simplified it. Earlier work had conjectured that the router needs to select at least 2 experts to learn properly, but Switch simplified the routing algorithm to reduce communication and computation costs. The authors named complexity, communication cost and training instability as the obstacles to MoE adoption, and by refining training techniques they trained large sparse models in lower precision such as bfloat16 for the first time. With the same compute resources, pre-training was up to 7x faster than models based on T5-Base and T5-Large, improvements were confirmed across all 101 languages, and scaling to 1 trillion parameters, they reported 4x faster pre-training than T5-XXL. There were gains even with only 2 experts, and distilling a large sparse model into a small dense model reduced model size by up to 99% while retaining 30% of the quality gain.

Mixtral paper is the case that brought MoE out of the lab. It uses the same architecture as Mistral 7B with 8 experts per layer and 2 used per token, and was trained with a 32k-token context. The authors stated that it outperformed or matched Llama 2 70B and GPT-3.5 on all evaluated benchmarks, and in particular far outperformed Llama 2 70B on mathematics, code generation and multilingual tasks. The Base and Instruct models were released under the Apache 2.0 license, making it a leading example of an open-weight MoE that anyone can download and use.

Where it is heading: how many, what shape, how to serve

The question in recent research has shifted from 'does MoE work?' to 'how should it be designed and operated?' The first issue is capacity allocation. A multilingual MoE scaling study paper proposed a scaling law that jointly accounts for active parameters, training tokens per language, number of experts and number of languages. It was fitted on 26,300 per-language loss observations from 64 pre-trained configurations, and reduced prediction error on unseen configurations by 28% compared with the strongest baseline. With active parameters and training tokens per language held equal, a model with 32 experts reduced the loss increase from expanding supported languages from 12 to 200 by 83% compared with a dense model. This means that the more languages a model covers, the more it benefits from having more experts.

The second issue is expert shape. Today's fine-grained MoE often makes each expert narrow while increasing the number of experts. An expert shape study paper argues that shape should not be decided as a by-product of the routing granularity. Treating the expert's input dimension and intermediate width as explicit design choices, it found that 'near-square' experts, where the two values are similar, were consistently better across experiments varying model scale and routing granularity. Allocating different numbers and shapes of experts per layer also lowered validation loss at the same parameter count, computation and communication.

The third issue is serving. DuoRate paper points out that MoE serving pays the precision cost twice. In storage, it pays for every expert the router could choose; in a single decoding step, it pays only for the experts actually chosen. The difference between the two is called the 'store-read gap'; in a public BitsMoE checkpoint, 2.54 bits per weight in storage became 7.50 bits in decoding traffic. DuoRate stores once and sets the number of bits read according to the task, and reported matching 4-bit quantization formats at 2.9 bits per weight on Qwen3-30B-A3B.

Limitations and open issues

First, less computation does not mean less memory. Even Mixtral paper, which computes only 13B per token, must keep all 47B parameters in memory because it does not know which experts a given token will call. This is exactly the issue DuoRate paper raises when it says 'storage cost is paid for every expert.' MoE therefore suits environments with 'plenty of GPU memory (VRAM) but a need to save compute,' and can actually be a burden on devices with little memory.

Second, load balancing and communication. If the router sends tokens to only a few experts, the remaining experts sit idle and the devices holding the overloaded experts become bottlenecks. Sparsely-Gated MoE paper noted from the start that a separate loss term may be needed to achieve the desired sparsity and load balance. When experts are spread across multiple devices, the network cost of exchanging tokens also grows. Switch Transformer paper naming complexity, communication cost and training instability as obstacles to adoption reflects the same point.

Third, efficiency comparisons require careful attention to the baseline. Claims that 'MoE is bigger and cheaper' are usually comparisons at the same active parameters or the same FLOPS. Measured by total parameters, memory or actual throughput (tokens/second), the conclusion can differ. Results that carry a condition such as 'with active parameters and training tokens held equal,' as in the multilingual study paper, should be read only within that condition.

Further study

If you are new to this, start with Section 2 of Switch Transformer paper. It summarizes router score computation, top-k selection and the weighted sum in two short equations, and includes a figure replacing a dense FFN with a Switch FFN, which helps you grasp the structure. Then read Section 1 of Sparsely-Gated MoE paper to learn why conditional computation had failed before (branching, shrinking batches, bandwidth, load balance), which makes the reasons behind MoE design decisions clear.

For a feel for a real model, read the abstract and architecture description of Mixtral paper and check what '2 of 8' and '47B vs. 13B' mean. Then choose recent research according to your interests. For model design, the expert shape study paper and the multilingual scaling study paper are good starting points; for deployment and cost, DuoRate paper.