Mistral
Mixtral 8x22B
MoE: all 8 experts resident in memory; only 2 active per token for compute.
Total params
141B
Active params
39B
Layers
56
Hidden size
6144
Attention heads
48
KV heads (GQA)
8
Vocab size
32,000
Native context window
65,536
Native precision
BF16
Experts (total)
8
Experts active / token
2
Size this model
Opens the sizing calculator pre-filled with this model at BF16 weights / FP16 KV cache and a typical workload — sign in to run it and see GPU/cloud recommendations.
Size this model →