DeepSeek-V3: an open high-end model from China at a very low training cost
In one sentence DeepSeek publishes V3, a 671B-parameter MoE (37B active), competitive with GPT-4o and Claude 3.5 Sonnet on many benchmarks. Full training: 2.788 million H800 GPU hours, about 5.6 million dollars at the stated rental cost.
On December 26, 2024 DeepSeek, a Chinese company based in Hangzhou, publishes the weights of DeepSeek-V3. It is a "mixture of experts" model with 671 billion total parameters, of which only 37 billion are used for each token. On the benchmarks published by DeepSeek it is on par with GPT-4o and Claude 3.5 Sonnet, and beats them in math and competitive coding.
The figure that sparks debate is the cost: the whole training took 2.788 million H800 GPU hours, which at the rental price assumed by DeepSeek (2 dollars per hour) comes to about 5.6 million dollars. The report specifies that this covers only the final training, not prior research and experiments; it is still far below what was thought necessary for a model of this level.
The weights are on Hugging Face, the code is MIT-licensed and the model allows commercial use.
Companies
DeepSeek
Tools
DeepSeek-V3
Tags
Sources