What is Mixture of Experts (MoE)?
TL;DR
Architecture that activates only a few specialist sub-networks per token, keeping massive total parameters affordable. Powers GPT-5, Gemini 3, and Mixtral.
Mixture of Experts (MoE): Definition & Explanation
Mixture of Experts (MoE) activates only a small subset of 'expert' sub-networks for each input token, allowing very large total parameter counts at low inference cost. Now mainstream in 2026 with GPT-5 (estimated 2–3T MoE), Gemini 3 Ultra, Mixtral 8x22B, DeepSeek V3, and Llama 4. A typical configuration has 1T+ total parameters but only tens-to-hundreds of billions activated per token, cutting inference compute by 5–10× vs. dense models. A learned routing layer dispatches tokens to experts, which tend to specialize in math, code, or natural language. Implementation challenges include (1) load balancing across experts, (2) communication cost in distributed training, and (3) fine-tuning stability — addressed by frameworks like GShard, Switch Transformer, and MegaBlocks. MoE is now standard for flagship APIs from Anthropic, OpenAI, and Google, while dense SLMs remain dominant on mobile and edge.