GPTQ: 3-4 bit post-training quantization for 175-billion-parameter GPT models
In one sentence Frantar, Ashkboos, Hoefler and Alistarh (IST Austria, ETH Zurich) publish GPTQ: one-shot 3-4 bit quantization without retraining, compressing a 175B model in about 4 GPU hours and running it on a single GPU.
Large language models like GPT-3 or OPT-175B take up hundreds of gigabytes and need several data-center GPUs just for inference. Quantization compresses the numbers representing the weights: instead of 16 bits per value, 3 or 4 are used.
The problem is that compressing this much usually degrades quality. GPTQ, published by researchers from IST Austria and ETH Zurich, uses second-order information to compensate for the errors introduced, layer by layer, on a small sample of calibration text. All without retraining the model.
The result: a 175-billion-parameter model is quantized in about four hours on one GPU and, once compressed, fits on a single data-center GPU (an 80 GB A100) with limited accuracy loss.
Companies
IST Austria, ETH Zurich, Neural Magic
Tools
GPTQ
Tags
Sources