What is Mixture of Experts?

In a mixture-of-experts (MoE) model, some layers of a transformer hold many expert networks. A router sends each token to only a few of them. So the total parameters (everything the model has learned) are far more than the active parameters (the part used for each token), which keeps inference cheaper. The idea goes back to a 1991 paper by Jacobs, Jordan, Nowlan and Hinton. Mistral's Mixtral 8x7B (December 2023) made it popular in open models: it has 8 experts and uses 2 per token. On 5 November 2026 Thinking Machines Lab launched Inkling 2 Small, an MoE model with 594 billion total and 17 billion active parameters. It said the weights will follow under an Apache 2.0 licence after safety testing. Compare models on AI models and their costs on LLM API prices.