Architecture
Mixture of experts (MoE)
An architecture where only a subset of the model's 'expert' sub-networks is used for each token, giving a large model the running cost of a smaller one.
A MoE model may have hundreds of billions of parameters in total but activate only a fraction per token. This is why some open models are huge on disk yet relatively cheap to run.
Frequently asked questions
What does Mixture of experts (MoE) mean?
An architecture where only a subset of the model's 'expert' sub-networks is used for each token, giving a large model the running cost of a smaller one. A MoE model may have hundreds of billions of parameters in total but activate only a fraction per token. This is why some open models are huge on disk yet relatively cheap to run.