llama.cpp: LLaMA 7B runs 4-bit on MacBook CPU
In one sentence Georgi Gerganov publishes llama.cpp, a dependency-free C/C++ implementation that runs 4-bit quantized LLaMA 7B on a Mac's CPU: one of the first practical ways to use an LLM of this size offline on a laptop.
Meta had just presented LLaMA, a family of language models designed to run on data-center GPUs. About two weeks later, Georgi Gerganov publishes llama.cpp: a program written in C/C++ that runs the smallest model, 7 billion parameters, on a MacBook's CPU.
The key is 4-bit quantization: instead of 16-bit floating-point numbers, each weight is approximated with 4 bits. Quality drops a little, but the model shrinks from about 13 GB to about 4 GB and becomes manageable on common hardware.
The author himself writes that he hacked it together in an evening and isn't sure it works fully correctly. But the message is clear: a language model of this size can run on anyone's laptop, with no internet and no server.
Companies
Georgi Gerganov (indipendente), Meta AI
Tools
llama.cpp, LLaMA
Tags
Sources