MoE Mixture of Experts
Model architecture with multiple "expert" networks — only a few activate per token, so large models become cheap to run.
Mixtral 8x7B = 8 experts at 7B each, but only 2 run at a time. Scalable: can have trillions of parameters without running them all every time. Powers modern large models.