Mixtral 8x7B: open-weights Mixture of Experts at GPT-3.5 level
In one sentence After a magnet link posted on X on December 8 with no explanation, on December 11 Mistral announces Mixtral 8x7B: Sparse MoE with 8 experts, 46.7B total parameters and 12.9B active per token, 32k context. Matches or beats GPT-3.5 and Llama 2 70B on most benchmarks, Apache 2.0 license.
Mistral makes a move that instantly becomes famous: on December 8 it posts a magnet-link torrent on X with zero explanation. Inside is Mixtral 8x7B, a Mixture of Experts (MoE) model: each layer has eight "expert" blocks, but only two are used per token. Result: 46.7B total parameters, but only about 12.9B do the work for each token.
Translated: it costs like a mid-size model but performs like a larger one. According to Mistral it beats Llama 2 70B on most benchmarks with 6x faster inference, and matches or beats GPT-3.5 on standard benchmarks.
The official post arrives on December 11. Apache 2.0 license: free commercial use. The same day Mistral opens its API platform in beta, where Mixtral powers the mistral-small endpoint.
Companies
Mistral AI
Tools
Mixtral 8x7B
Tags
Sources