StarCoder2 and The Stack v2: open code models at 3B, 7B and 15B
In one sentence BigCode releases StarCoder2 in three sizes (3B, 7B, 15B), trained on 3.3-4.3 trillion tokens from The Stack v2, built with Software Heritage on repositories spanning 619 languages. The 15B matches or beats CodeLlama-34B. BigCode OpenRAIL-M license.
StarCoder2 is the second generation of code models from the BigCode project, led by Hugging Face and ServiceNow. It comes out on February 28, 2024 in three sizes: 3, 7 and 15 billion parameters.
The data comes from The Stack v2, built from the Software Heritage archive, which collects source code in 619 programming languages. The training set is about four times larger than the first StarCoder's and also includes GitHub pull requests, Kaggle notebooks and documentation.
The largest model, trained by NVIDIA, matches or beats CodeLlama-34B, which has more than twice as many parameters. For transparency, the Software Heritage identifiers of the code used are published.
Companies
BigCode, Hugging Face, ServiceNow, NVIDIA
Tools
StarCoder2
Tags
Sources