Solar 10.7B: depth upscaling to merge layers from two LLaMA-2 models

Solar 10.7B comes from an unusual idea: take two fine-tuned LLaMA-2 models and combine them by joining their internal layers, like building a taller structure by taking floors from two different buildings.

This technique, called depth upscaling, produces a 10.7-billion-parameter model that performs better than the two original models separately.

Upstage's model won the open-source OpenLLM leaderboard in November 2023, proving competitive with models of much larger size.