Solar 10.7B: Upstage depth-scales a 7B model and tops the Open LLM Leaderboard
In one sentence Upstage releases Solar 10.7B, built with depth up-scaling from Mistral 7B weights: on 14 December 2023 it announces first place on the Hugging Face Open LLM Leaderboard with an average of 74.2. The base model is licensed under Apache 2.0.
Solar 10.7B comes from a simple idea: take a good 7-billion-parameter model, duplicate it and stack the two copies, removing some layers at the seam, to get a deeper model. Then retrain it so the two halves work well together.
South Korea's Upstage calls this technique depth up-scaling. The result has 10.7 billion parameters and, according to the 14 December 2023 announcement, reaches first place on the Hugging Face Open LLM Leaderboard with an average score of 74.2.
The base model is released under the Apache 2.0 license, which allows commercial use; an Instruct version is also available.
Companies
Upstage
Tools
SOLAR-10.7B
Tags
Sources