What Is GGUF Quantization for Local AI Models?
GGUF quantization is the process of storing a language model's weights in fewer bits than the original training precision, so the model file becomes smaller and the model fits in less memory on a personal computer. The trade-off is precision: a quantized model is an approximation of the original, and lower bit counts save more memory while introducing more error into the model's answers.
Updated 2026-11-02. This page is general information, not legal advice.
Why quantization exists at all
Models are trained in high precision, typically 16-bit or 32-bit floating point numbers, and a model with billions of parameters is enormous at that precision. A personal computer cannot always hold the full-precision model alongside the operating system and other applications. Quantization solves the storage problem by rounding each weight to a shorter representation, for example 8-bit or 4-bit values, and storing a scale factor that lets the inference engine reconstruct an approximation of the original value when it runs. The result is a model that is a fraction of the original size and runs on ordinary hardware.
What GGUF is
GGUF is the file format used by llama.cpp for storing models, including their quantized weights. A GGUF file carries the model's architecture, its weights in the chosen quantization, and metadata such as the tokenizer, so a single file is enough to run the model. When you download a model file whose name ends in Q4_K_M or Q8_0, you are downloading a GGUF file quantized with that preset, and the suffix tells you how aggressively it was compressed.
The common presets, and what they mean
- Q8_0: 8-bit quantization, the largest common option and the closest to the original model, for machines with memory to spare.
- Q6_K: 6-bit quantization, a step down in size with quality close to the original for most uses.
- Q5_K_M: 5-bit quantization, a middle ground that balances size and quality for everyday use.
- Q4_K_M: 4-bit quantization, the widely used default for personal machines, because it roughly quarters the model size while keeping answers useful.
- Q3_K_M and Q2_K: 3-bit and 2-bit options for tight memory limits, with noticeably more quality loss.
The "K" in these names refers to a family of quantization methods that group weights into blocks and choose scales per block, which preserves quality better than rounding every weight with a single scale. The "_M" suffix marks a medium variant within that family, as opposed to a smaller or larger sibling.
How to quantize a model yourself
The llama.cpp workflow has two stages. First, convert the original model into a GGUF file at full precision with the project's conversion script. Second, run the quantization tool on that file with the preset you want, which produces a new, smaller GGUF file. For better quality at low bitrates, you can first build an importance matrix from calibration text, which measures which weights matter most, and then pass that matrix to the quantization tool so it spends its limited bits where they count. Each stage is a command, and each output is a file you can test before committing to it.
How to choose a preset
Start with Q4_K_M unless you have a reason not to. If answers show clear degradation on your tasks, step up to Q5_K_M or Q6_K. If the model will not fit in memory, step down to Q3_K_M and test again, accepting that some tasks will suffer. The right preset is the one that fits your machine and passes your own tasks, not the one a benchmark favors, because quality loss shows up unevenly across different kinds of questions.
Frequently asked questions
Does quantization make a model worse?
Yes, to some degree. Lower bit counts introduce more approximation error, though the effect varies by task and is small at moderate presets like Q4_K_M or Q5_K_M.
What does Q4_K_M mean in a model filename?
It is a GGUF quantization preset: roughly 4-bit weights using the K-quant method, medium variant, produced by the llama.cpp quantization tool.
Can I requantize an already quantized model?
The tool allows it, but it is not recommended, because each round of rounding adds error on top of the previous round.