Mixture of Experts (MoE)
A neural network architecture where multiple specialized 'expert' sub-networks exist, with a router selecting which to use for each input. Enables massive models at lower compute cost.
Mixture of Experts (MoE) is a neural network architecture pattern where a model contains many specialized sub-networks ("experts"), and a learned router selects which 1-2 experts to use for each input token. Only the chosen experts perform computation — the rest stay idle.
This means an MoE model can have far more total parameters than a dense model with similar compute cost. DeepSeek V3 (671B total parameters, 37B active) and Mixtral 8x22B use this approach. Major frontier models (GPT-4, Claude 4) are believed to be MoE.
Benefits: more capacity for the same inference cost; specialized experts for different domains. Drawbacks: more complex training; routing decisions can be brittle. MoE has become the dominant architecture pattern for frontier models since 2024.