High Foundation Models · 1 min read
Falcon 40B: first open-weight model to beat LLaMA 65B
In one sentence The Technology Innovation Institute UAE releases Falcon 40B: trained on 1T tokens of RefinedWeb, it beats LLaMA 65B on benchmarks with a commercial license.
Reading level
Falcon 40B is a language model developed by a research institute in the UAE. It was released with a license allowing commercial use, which was rare for such a powerful model in 2023.
Its distinguishing feature is data quality: the team created RefinedWeb, a dataset of one trillion filtered and deduplicated tokens from the web, much cleaner than collections used by competitors.
The result was immediate: Falcon 40B topped the open-source leaderboards, beating Meta's LLaMA 65B despite having fewer parameters.
Companies
Technology Innovation Institute
Tools
Falcon-40B
Tags
FalconOpen WeightsTIIRefinedWebScaling
Sources