Skip to content
AImpact
IT EN
Inference Intermediate Also known as: Quantizzazione

Quantization

A technique that reduces the numeric precision of model weights (for example from 16 to 4 bits) so it takes less memory and runs faster.

ShareLinkedInX

In practice

It is what lets you run a Llama 70B on a single GPU or a 7B model on a Mac. You lose a bit of quality but often not much. Typical tools: GGUF, AWQ, GPTQ. Useful for on-prem or edge deployment.

Related terms

Inference compute LoRA

Seen in the wild

11 entries mentioning it
  1. Usable 2-bit quantization: frontier reasoning models drop below 32GB RAM
    Medium
  2. torchao: PyTorch-Native Quantization and Sparsity Without Custom CUDA
    Medium
  3. KV Cache Quantization FP8/INT8: Double User Density per GPU
    High
  4. bitsandbytes 0.43: QLoRA and NF4/FP4 quantization for 4-bit fine-tuning
    Medium
  5. LLM Compressor: unified toolkit for quantization and sparsity with native vLLM integration
    Medium
  6. GGUF specification: the standard format for local quantized LLM models
    Medium
  7. ExLlamaV2: high-speed quantized LLM inference on consumer GPUs
    Medium
  8. llama.cpp K-quants: the intelligent quantization that transformed local models
    High
  9. AWQ: activation-aware 4-bit quantization for edge deployment with accuracy above GPTQ
    High
  10. llama.cpp: LLaMA 7B runs 4-bit on MacBook CPU
    Landmark
  11. GPTQ: 4-bit post-training quantization making GPT-scale inference practical
    High
← All terms