Mixture of Experts Routing: How MoE Models Decide Which Experts Fire

A Mixture of Experts (MoE) model is only as good as its router — the small network that decides which of hundreds of expert sub-networks gets to process each token. Get routing wrong and half your experts sit idle while the rest overheat, a failure mode called expert collapse. DeepSeek-V3, a 671-billion-parameter model with 256 experts per layer, solves this with a bias-adjustment trick that never touches the training loss directly. This piece unpacks exactly how that works, and how three earlier routing algorithms tried and partly failed to solve the same problem.

The problem routing exists to solve

Every Transformer layer in a dense model runs its entire feed-forward network (FFN, the block of matrix multiplications that processes each token's vector representation) on every token. A Mixture of Experts layer swaps that single FFN for N smaller FFNs — the "experts" — plus a router that decides which subset of experts each token gets sent to, typically two out of eight, or eight out of 256. Only the selected experts do any work on that token.

This is where routing stops being a footnote and becomes the entire engineering problem. If routing were random, MoE would just be a slower, more complicated dense network. The router has to learn, from scratch, which tokens belong with which experts, all while training a language model that has never seen either concept before. Get the router's incentives wrong, and the system converges to something worse than dense: a handful of "popular" experts absorb almost every token, the rest of the network's capacity goes unused, and you have effectively built an expensive small model.

Google's original Switch Transformer paper found that without a corrective loss term, routing collapses so hard that a small number of experts absorb nearly all tokens within the first few thousand training steps — the rest never receive enough gradient signal to learn anything useful ([Fedus, Zoph & Shazeer, 2021](https://arxiv.org/pdf/2101.03961)).

Think of the router like a hospital's triage desk. If triage always sends patients to the two doctors nearest the door regardless of specialty, those two doctors burn out from overload while twenty specialists down the hall never see a single case. The hospital paid to train and staff twenty specialists but functionally operates as a two-doctor clinic. Load balancing is the triage protocol that forces patients toward the right specialist and prevents that collapse.

Token-choice routing: the default, and its two failure modes

The routing scheme almost every MoE model starts from is called token-choice, or top-K routing. It was formalized by Noam Shazeer's team at Google in the 2017 paper that revived MoE for deep learning, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (Shazeer et al., 2017). Here is the mechanism step by step.

1. The router is a single linear layer. It takes a token's hidden-state vector and multiplies it by a learned weight matrix, producing one score per expert.
2. A softmax converts those scores into a probability distribution over all N experts.
3. The router keeps only the top K experts by score (K is usually 1 or 2) and discards the rest.
4. Each selected expert processes the token independently, and their outputs are combined in a weighted sum using the router's own softmax scores as weights.

That is the entire mechanism — a few lines of matrix math. The complexity is not in the routing formula. It is in what happens when you let that formula train unsupervised on billions of tokens, because two failure modes emerge immediately.

Why does MoE routing collapse without a balancing loss?

Routing collapses because early in training, one expert's random initialization makes it marginally better than the others at a common token type, so the router sends it slightly more traffic. That expert then gets more gradient updates, gets better still, and attracts even more traffic — a rich-get-richer feedback loop. Within a few thousand steps, most experts stop receiving meaningful gradient signal and never specialize at all.

The second failure mode is a hardware problem, not a training-dynamics problem: capacity overflow. In distributed training, each expert lives on a fixed-size hardware buffer sized for roughly (total tokens ÷ number of experts) with some slack. If routing sends more tokens to one expert than its buffer can hold, the overflow tokens get dropped — they skip that layer's expert computation entirely and pass through via a residual connection, per the Switch Transformer paper's account of token dropping (Fedus, Zoph & Shazeer, 2021). Dropped tokens are trained on incomplete information, which degrades quality in ways that are hard to trace back to their root cause.

Switch Transformer's fix: an auxiliary loss that punishes imbalance

Google's Switch Transformer (2021) is the paper that made top-1 routing (K=1, the simplest possible case) work reliably at scale, and it did so by adding a second loss term to training, separate from the main next-token-prediction loss. This auxiliary load-balancing loss compares two quantities per expert: the fraction of tokens actually routed to it, and the average routing probability the router assigned to it across the batch. The loss penalizes the model whenever those two numbers diverge, which happens exactly when routing is uneven (Fedus, Zoph & Shazeer, 2021).

The auxiliary loss is scaled by a small weight, typically around 0.01, and simply added to the main training loss before backpropagation. Get that weight too low and load balancing barely happens; get it too high and the router starts making bad routing decisions purely to satisfy the balance term instead of to maximize prediction quality. Tuning that single hyperparameter across model scales became its own minor sub-field within MoE engineering.

Switch Transformer also introduced the capacity factor, a multiplier above 1.0 that gives each expert's buffer some headroom above the perfectly-even allocation, and a router z-loss that discourages the router's raw scores from growing too large, which independently improves training stability. Together these three levers — the balancing loss, the capacity factor, and the z-loss — are what let Switch Transformer scale past a trillion parameters with routing that mostly held together, though never perfectly.

Expert-choice routing: flip who picks whom

In 2022, a separate Google Research team asked an obvious but under-explored question: what if the load-balancing problem is solved more directly by not letting tokens choose experts at all? Their answer was expert-choice routing, published as "Mixture-of-Experts with Expert Choice Routing" at NeurIPS 2022 (Zhou et al., 2022).

In expert-choice routing the direction of selection is reversed. Instead of each token picking its top-K experts, each expert picks its own top-K tokens from the batch, based on the same underlying router scores. Since every expert is guaranteed to select exactly a fixed number of tokens — no more, no less — perfect load balance is enforced by construction. There is no auxiliary loss to tune and no dropped-token buffer overflow, because the math cannot produce an imbalanced outcome in the first place.

The tradeoff is that expert-choice routing changes what the system is doing at inference time. A token can now be selected by zero experts (if it scores low with every expert relative to other tokens in the batch) or by many experts, and the number of experts a given token receives depends on the rest of the batch it happens to be processed alongside. That batch-dependence makes expert-choice routing awkward for autoregressive text generation, where you generate one token at a time and don't have a batch of future tokens to compare against — which is a large part of why it has seen more adoption in research and encoder-style settings than in production decoder-only chat models.

DeepSeek-V3's auxiliary-loss-free approach

By late 2024, the dominant open-weight approach had shifted again. DeepSeek-V3, a 671-billion-parameter model that activates 37 billion parameters per token across 256 routed experts plus one always-active shared expert, introduced a routing fix that keeps token-choice routing's simplicity but removes the auxiliary loss entirely (DeepSeek-AI, DeepSeek-V3 Technical Report, December 2024).

The mechanism: each expert gets a learnable bias term. That bias is added to the expert's raw routing score when the router decides which experts to select for a token — but critically, it is excluded from the softmax weights used to combine expert outputs. After each training step, the bias for any expert that received more than its fair share of tokens gets nudged down, and the bias for any under-loaded expert gets nudged up. Over many steps this pushes the selection process toward balance without ever touching the model's actual gradient through a competing loss function.

Why does removing the auxiliary loss matter?

An auxiliary loss creates two competing training objectives: predict the next token accurately, and keep routing balanced. Whenever those objectives conflict, the model has to trade off some prediction quality for balance. DeepSeek-V3's bias-only approach keeps the balancing decision entirely inside the discrete selection step, so the gradient that flows back through the network is purely about next-token prediction. DeepSeek-AI reports this produces better downstream performance than equivalent models trained with a traditional auxiliary loss (DeepSeek-AI, 2024).

DeepSeek-V3 pairs this with fine-grained experts — 256 relatively small experts instead of the 8 or 16 large experts used by Mixtral or DBRX — plus a shared expert that processes every token regardless of routing, intended to absorb common patterns so the routed experts can specialize more narrowly on rarer patterns. DeepSeek's earlier V2 paper documented the fine-grained-expert design that V3 builds on (DeepSeek-AI, DeepSeek-V2 Technical Report, 2024).

A worked example of the bias-adjustment update

The bias mechanism is simple enough to trace through with made-up but representative numbers, using the update rule DeepSeek-AI describes in the V3 technical report (DeepSeek-AI, 2024): after each training step, compare how many tokens an expert actually received against its fair share, and nudge that expert's bias in the opposite direction by a small fixed step size.

Say a training batch contains 8,192 tokens, DeepSeek-V3 selects 8 of its 256 routed experts per token, and every expert would receive exactly its fair share of traffic if selection were perfectly even. Total routing decisions in the batch equal 8,192 tokens multiplied by 8 selections each, or 65,536 expert-token assignments. Spread evenly across 256 experts, the fair share per expert is 65,536 divided by 256, which is 256 tokens per expert, per batch.

Now suppose Expert 17 is an early favorite: instead of its fair share of 256 tokens, it draws 340 tokens in that batch, an overload of 84 tokens above target. Expert 203, by contrast, draws only 190 tokens, a shortfall of 66 tokens below target. Under DeepSeek-V3's update rule, Expert 17's bias term is decreased by the fixed step size γ (the paper uses a small constant, updated additively rather than proportionally to the size of the imbalance) and Expert 203's bias is increased by that same step size. Because the bias is added directly to the raw routing score used for top-K selection — not to the softmax weights that scale each expert's output — the very next batch, Expert 17's score for borderline tokens drops slightly, making it lose out on some assignments it would otherwise have won, while Expert 203's score rises slightly, letting it win assignments it would otherwise have lost.

Run that correction over thousands of steps and the effect compounds: any expert that stays chronically overloaded keeps losing bias, and any expert that stays chronically underloaded keeps gaining it, until traffic settles near the 256-tokens-per-expert equilibrium without a single gradient update ever having been computed from a balancing loss term. The correction happens entirely in the discrete selection step, between one forward pass and the next, which is why it does not compete with the next-token-prediction objective the way Switch Transformer's auxiliary loss does.

Two more routing variants: hashing and soft assignment

Token-choice and expert-choice are not the only ways to solve this problem. Two other schemes take almost opposite approaches, and both are worth understanding because they show the outer edges of the design space.

Hash routing: throw out the router entirely

Hash Layers, introduced by Meta AI researchers in 2021, delete the learned router altogether (Roller et al., 2021). Instead of a neural network scoring which expert a token should go to, each token is assigned to an expert using a deterministic hash function applied to the token's identity, similar to how a hash table assigns a key to a bucket. Because the hash function is fixed before training even starts, routing is perfectly balanced by design — no auxiliary loss, no bias terms, no capacity overflow, because the assignment counts are known in advance and can be engineered to be even.

The Hash Layers paper reported that this non-learned routing was competitive with, and in some configurations outperformed, learned routing methods like Switch Transformer's top-1 gating, while requiring zero routing parameters and no extra training-loss terms (Roller et al., 2021). The catch is that hash routing cannot adapt: it cannot learn that, say, code tokens and poetry tokens benefit from different experts, because the assignment was decided by a hash function before the model ever saw a training example. It works because Transformer layers are flexible enough to compensate for an arbitrary but fixed partition of the vocabulary, not because the partition itself is meaningful. That is why hash routing has stayed mostly a research curiosity and an ablation baseline rather than a production choice for frontier models — it proves routing can be simplified further than anyone expected, but it gives up the specialization benefits that make MoE attractive in the first place.

Soft MoE: skip discrete selection altogether

Google DeepMind's Soft MoE, published at ICLR 2024, attacks the problem from a different angle: instead of routing tokens to experts, it routes weighted combinations of all tokens to each expert (Puigcerver, Riquelme, Mustafa & Houlsby, "From Sparse to Soft Mixtures of Experts," 2024). Every expert receives a small number of "slots," and each slot is filled with a learned weighted average across every token in the input, not a hard assignment of individual tokens. Because the combination weights are produced by a differentiable softmax rather than a discrete top-K selection, there is no token dropping, no capacity overflow, and no need for an auxiliary balancing loss — the math cannot produce an imbalanced outcome because no token is ever exclusively "claimed" by one expert.

The tradeoff is architectural rather than statistical. Soft MoE's slot-based averaging assumes access to a full batch or sequence of tokens at once to compute those weighted combinations, which fits vision Transformers and encoder-style processing comfortably but maps awkwardly onto autoregressive decoding, where tokens are generated one at a time and future tokens do not exist yet when you need to route the current one. The paper's headline result — a Soft MoE Huge/14 model with over 40 times more parameters than a dense ViT-Huge/14, at only about 2% higher inference time and clearly better quality on image classification benchmarks (Puigcerver et al., 2024) — is why Soft MoE has found its strongest adoption in vision and multimodal encoders rather than in the decoder-only chat models that dominate the MoE conversation today.

Comparing the five routing approaches

Routing method	Who chooses	How balance is enforced	Main tradeoff	Used by
Token-choice + auxiliary loss	Each token picks top-K experts	Extra loss term penalizes uneven expert utilization	Balance loss competes with prediction accuracy; needs careful weight tuning	Switch Transformer, Mixtral, GShard
Expert-choice routing	Each expert picks its top-K tokens	Fixed per-expert token budget, balance is structural	Token count per expert varies with batch; awkward for one-token-at-a-time generation	Research and encoder-style models
Hash routing	A fixed hash function, not a learned network	Assignment is deterministic and pre-balanced before training starts	No adaptive specialization; the partition is arbitrary, not learned	Meta's Hash Layers, research baselines
Soft MoE	No discrete choice — every expert sees a weighted blend of all tokens	Differentiable combination weights, no hard assignment to overflow	Needs a full batch/sequence at once; awkward for autoregressive decoding	Vision Transformers, multimodal encoders
Auxiliary-loss-free bias adjustment	Each token picks top-K experts	Per-expert bias term updated after each step, outside the loss	Adds a bias-update rule to implement and tune, though no competing loss	DeepSeek-V3, DeepSeek-R1

MoE vs. dense: what routing costs you at inference time

Everything above is about training-time routing quality, but routing has a second, separate life once the model is deployed: it directly shapes inference cost and latency, and the tradeoffs there run in the opposite direction from what training-time intuition suggests.

At small batch sizes — the common case for a single user chatting with a model — MoE inference latency is dominated not by compute but by memory bandwidth: the time it takes to read the selected experts' weights out of GPU memory. Microsoft's DeepSpeed-MoE team frames this directly, and reports that a well-engineered MoE inference system can reach up to 4.5 times lower latency and 9 times lower cost than a quality-equivalent dense model at comparable quality (Rajbhandari et al., "DeepSpeed-MoE," 2022). That gain exists because a token only ever touches a handful of experts — DeepSeek-V3 activates 37 billion of its 671 billion total parameters per token — so the actual matrix multiplications for that token are far cheaper than running a dense model with the equivalent parameter count.

The catch is that "far cheaper compute" and "far cheaper deployment" are not the same claim. Every expert an MoE model owns, whether a given batch routes tokens to it or not, still has to sit resident in GPU memory in case it is needed, because you cannot know in advance which experts the next request will need. That is why a sparse MoE model's memory footprint can run several times higher than a dense model with the same active-parameter count: one comparison found an MoE configuration needing roughly 18.9 GB per GPU against 2.2 GB for a dense counterpart, an increase of about 8.6 times (Introl, "Mixture of Experts Infrastructure," 2026). Dense models trade that away: they compute more per token, but every one of their parameters is doing useful work on every token, so nothing sits idle in memory waiting for a routing decision that may never arrive.

Batch size changes the picture again. At light load, only a small slice of an MoE model's total experts get activated across the batch, so memory traffic and compute both stay low relative to a dense model of similar quality — that is where the biggest latency wins show up. As concurrent traffic grows, however, a wider and wider spread of experts gets touched across the larger batch, and those savings erode; at high enough concurrency, MoE serving cost can approach dense-model territory, because nearly every expert ends up loaded and active somewhere in the batch anyway (Huang et al., "Toward Efficient Inference for Mixture of Experts," NeurIPS 2024).

There is a third cost that dense models never face at all: cross-GPU communication. Because a 671-billion-parameter model's experts are typically split across many GPUs — no single GPU has enough memory to hold all 256 experts per layer — a token's selected experts often live on different physical devices than the token itself. Routing a token to its chosen experts and gathering their outputs back requires an "all-to-all" communication step across the GPU interconnect on every MoE layer, and at scale that interconnect traffic, not raw FLOPs, becomes the binding constraint on throughput (Introl, 2026). This is the direct, practical reason serving frameworks like vLLM and SGLang invest in dedicated expert-parallel dispatch kernels: a routing decision that looked free in the training loss function is not free once it has to cross a physical network link.

Put together, the honest summary is that MoE trades a training-time and light-load inference win for a standing memory tax and a communication cost that dense models simply do not carry. Whether that trade nets out in an MoE model's favor depends heavily on batch size, GPU topology, and how many concurrent users a deployment actually serves, which is why the same underlying architecture can look either dramatically cheaper or barely different from dense, depending entirely on the serving conditions it runs under.

Why this matters beyond training stability

Routing decisions ripple into every downstream cost of running an MoE model. Poor load balancing during training means some experts underfit, which shows up later as inconsistent quality: the same model can answer one prompt sharply and stumble on a superficially similar one, because the token happened to route to a weaker expert. Researchers studying MoE behavior have documented that routing-induced inconsistency as a known characteristic of these architectures, distinct from the randomness of sampling temperature.

Routing also determines serving cost directly. Every expert, whether it receives one token or one million, has to be loaded into GPU memory for a batch that might route to it — this is why a 47-billion-parameter MoE model needs roughly the same VRAM as a 47-billion-parameter dense model even though it only computes with a fraction of those parameters per token. Balanced routing during training does not reduce that memory floor, but it does mean the expert-parallel communication pattern at inference is more predictable, which is why inference engines such as vLLM and SGLang build dedicated kernels around expert-parallel token dispatch rather than treating MoE layers as an afterthought.

Where load-balancing research is headed

Expert-choice routing solved balance but broke autoregressive generation's batch independence assumption. DeepSeek-V3's bias trick solved the competing-objective problem but still relies on token-choice routing's basic vulnerability to per-token popularity swings within a batch. Neither is a complete solution, and both remain active research areas as of 2026, with follow-up work exploring hybrid schemes, dynamic expert counts per token, and finer-grained expert pools that reduce how costly any single routing mistake becomes.

The auxiliary-loss-free approach that DeepSeek-V3 popularized has already been adopted or adapted by several open-weight MoE releases since, largely because it removes one of the trickiest hyperparameters — the auxiliary loss weight — from the training recipe entirely. That is a meaningful engineering simplification even before accounting for the reported quality gains, and it is a reasonable bet that most new frontier MoE models trained after 2024 will start from some variant of it rather than the original Switch Transformer recipe.

Common misunderstandings about MoE routing

"More experts always means better load balancing." The opposite is often true. More experts means more ways for the router to develop a lopsided preference early in training, and finer-grained experts (DeepSeek-V3's 256 small experts versus Mixtral's eight large ones) need more careful balancing machinery, not less, even though they enable better specialization once balance is achieved.

"Load balancing is only a training-time concern." It also governs inference latency. If a batch of real user requests happens to route unevenly — plausible, since real traffic is not IID with training data — some experts become a bottleneck while others sit idle, and inference engines have to handle that skew with techniques like expert replication across GPUs.

"Expert-choice routing is strictly better than token-choice." It solves load balance structurally, but the batch-dependent token count per expert makes it a poor fit for standard autoregressive decoding, where you generate one token at a time without a batch of alternatives to compare against. That is a real architectural constraint, not a minor implementation detail.

"The router 'knows' what each expert specializes in." The router only ever sees a token's hidden-state vector and produces scores; expert specialization is an emergent statistical pattern that arises from training, not something explicitly assigned. Researchers have observed experts that fire disproportionately on code tokens or specific languages, but no one designed that division of labor in advance.

Frequently asked questions

Q: What is a Mixture of Experts model in simple terms? A: A Mixture of Experts (MoE) model replaces one large feed-forward network per Transformer layer with many smaller "expert" networks plus a router. For each token, the router activates only a handful of experts — commonly 2 out of 8, or 8 out of 256 — instead of running every parameter on every token, which is what a dense model does (Shazeer et al., 2017).

Q: How is MoE different from a dense transformer? A: A dense Transformer runs its full parameter count on every token, so total parameters and active parameters are the same number. An MoE model separates those two numbers: DeepSeek-V3 has 671 billion total parameters but activates only 37 billion per token, because routing sends each token to a small subset of experts rather than the whole network (DeepSeek-AI, 2024).

Q: What is expert collapse in MoE routing? A: Expert collapse is the failure mode where a router develops an early, self-reinforcing preference for a small number of experts, starving the rest of gradient signal. Google's Switch Transformer team found that without a corrective loss term, this collapse happens within the first few thousand training steps and leaves most experts undertrained (Fedus, Zoph & Shazeer, 2021).

Q: What is the difference between token-choice and expert-choice routing? A: In token-choice routing, each token selects its top-K experts based on router scores, and load balancing has to be enforced externally, usually with an auxiliary loss. In expert-choice routing, each expert instead selects its own top-K tokens from the batch, which guarantees even load by construction but makes the number of experts a given token receives depend on the rest of the batch (Zhou et al., NeurIPS 2022).

Q: How does DeepSeek-V3 balance experts without an auxiliary loss? A: DeepSeek-V3 gives each expert a learnable bias term that is added only to the routing-selection score, not to the softmax weights used to combine expert outputs. After every training step, overloaded experts get their bias nudged down and underloaded experts get theirs nudged up, gradually steering token assignment toward balance without ever competing with the next-token-prediction loss (DeepSeek-AI, 2024).

Q: What is hash routing in Mixture of Experts? A: Hash routing assigns each token to an expert using a fixed, non-learned hash function decided before training starts, rather than a trained router network. Meta's Hash Layers paper found this approach could match or beat learned top-1 routing while needing zero routing parameters and no balancing loss, at the cost of being unable to adapt which tokens go where based on what the model learns (Roller et al., 2021).

Q: What is Soft MoE and how does it avoid token dropping? A: Soft MoE, from Google DeepMind, has each expert process a weighted blend of every token in the input rather than a hard, discrete selection of individual tokens. Because the routing weights are produced by a fully differentiable softmax instead of a top-K cutoff, no token is ever fully excluded or dropped for capacity reasons, which removes both the auxiliary loss and the token-dropping problem at once (Puigcerver et al., ICLR 2024).

Q: Does MoE make inference faster than a dense model of the same quality? A: Often, but not unconditionally. At small batch sizes, MoE inference can be several times cheaper and lower-latency than an equivalent dense model because only a few experts get activated per token; DeepSpeed-MoE reports up to 4.5 times lower latency and 9 times lower cost in that regime (Rajbhandari et al., 2022). At high concurrency, more of the total expert pool gets touched across the larger batch, and the savings shrink.

Q: Why does an MoE model need more GPU memory than its active parameter count suggests? A: Every expert has to sit loaded in GPU memory in case the next batch routes tokens to it, even if it received zero tokens in the current batch, because there is no way to know in advance which experts will be needed. That standing memory requirement is why one MoE-versus-dense comparison found the MoE model needing roughly 8.6 times more memory per GPU than a dense model with a comparable active-parameter footprint (Introl, 2026).

Q: Which models use Mixture of Experts architecture? A: Production and open-weight MoE models include Google's Switch Transformer and GLaM, Mistral's Mixtral 8x7B and 8x22B, Databricks' DBRX, and DeepSeek's V2, V3, and R1 models, which activate 37 billion of 671 billion total parameters per token using the auxiliary-loss-free bias-adjustment routing described above (DeepSeek-AI, 2024).

The takeaway

MoE routing is not a solved problem with one canonical answer — it is a live tradeoff between three imperfect strategies, each optimizing for a different failure mode. Token-choice routing with an auxiliary loss is simple but forces a compromise between balance and accuracy. Expert-choice routing enforces balance structurally but assumes a batch structure that autoregressive generation does not naturally have. DeepSeek-V3's bias-adjustment method removes the competing-loss compromise while keeping token-choice routing's generation-friendly structure, which is likely why it has become the reference approach for new frontier-scale open-weight models. Whichever scheme a model uses, the router's job is the same: decide, correctly and fairly, which of hundreds of specialists actually gets to work on each word.

---

Read next:

• How Do Neural Networks Work? The Architecture Behind Modern AI
• LLM Quantization — Run Models Locally
• Test-Time Compute Explained — LLM Reasoning
• AI Training vs. Inference: What Actually Happens When AI Learns and Thinks