What is a GGUF model file, and how does it work?

Updated November 3, 2026.

A GGUF file is a binary file format for storing a model for inference. The GGUF specification calls it a format for storing models for inference with GGML and executors based on GGML, designed for fast loading and saving of models and for ease of reading. A model is usually developed in PyTorch or another framework and then converted to GGUF for use in GGML. The format is a successor to GGML, GGMF and GGJT, and it is designed to be unambiguous because the file contains all the information needed to load the model.

What is inside a GGUF file

The specification lists the goals of the format: single-file deployment, extensibility, compatibility with mmap for fast loading, ease of use, and full information, meaning no extra settings need to be supplied by the user. In practice the file begins with a header, then a block of key-value metadata, then a list of tensor descriptions, then padding, then the tensor data itself. The header holds a magic number, a version, the number of tensors and the number of metadata pairs. Tensor data is aligned using the value in the general.alignment field, and the specification says to assume 32 if a writer left it out. Files are little-endian unless stated otherwise.

The metadata is what makes the file self-describing. The specification defines general fields such as general.architecture, general.name, general.license and general.file_type, tokenizer fields such as tokenizer.ggml.tokens, and a tokenizer.chat_template field that holds a Jinja template for the input format the model expects. When any tensor is quantized, the file must also record general.quantization_version. Hugging Face's GGUF page makes the same point from the user side: unlike tensor-only formats such as safetensors, GGUF encodes both the tensors and a standardized set of metadata, and the Hub has a viewer for that metadata and the tensor information.

Reading a GGUF file name

GGUF has a naming convention of base name, size label, fine-tune, version, encoding, type and shard, ending in .gguf. The specification says this is meant to help people see the main details at a glance and is not intended to be perfectly parsable, because real file names vary. Its examples include Hermes-2-Pro-Llama-3-8B-F16.gguf, where F16 is the weight encoding, and Grok-100B-v1.0-Q4_0-00003-of-00009.gguf, which is shard 3 of 9 for a Q4_0 encoding. A name such as mtp-Qwen3-27B-v1.0-Q4_K_M.gguf shows the Q4_K_M label too. The file_type list in the specification includes entries named MOSTLY_Q4_K_M and MOSTLY_Q8_0, which is where those labels come from. The specification does not explain every suffix in plain words, so read the model card for what a label means for a given file.

Quantization labels and size

Hugging Face's GGUF page lists the quantization types. F32 is a 32-bit float and F16 a 16-bit float. Q4_K is described as 4-bit quantization in super-blocks of 8 blocks with 32 weights each, resulting in 4.5 bits per weight. Q4_0, Q5_0 and Q8_0 are described as legacy round-to-nearest methods that are not used widely today. The llama.cpp README says the project supports 1.5-bit through 8-bit integer quantization for faster inference and reduced memory use. Fewer bits per weight means a smaller file, and the exact effect on answer quality depends on the model, so test the file you plan to use. Our page on how much GPU memory a model needs covers fitting a file into memory.

Loading a GGUF file in Ollama

Ollama's import page says to create a Modelfile containing FROM /path/to/file.gguf and then run ollama create with a name for the model. Ollama does not quantize GGUF models during import, so quantize first with a GGUF tool such as llama.cpp's llama-quantize. For a split model, keep the original split file names and point FROM at a wildcard such as /path/to/model-*.gguf that matches every shard. The llama.cpp README shows another route: its quick start downloads and runs a model straight from Hugging Face by passing a repository name that ends in -GGUF.

Ollama's quickstart says local models run on your computer and need no API key. Once the file is on disk, where it lives and how large it is are separate questions, and the page on storage for model files covers them.

Frequently asked questions

Is GGUF the same as quantization?

No. GGUF is the file format, and quantization is how the weights inside it are stored. The Hugging Face GGUF page lists full-precision types such as F32 and F16 alongside quantized types such as Q4_K and Q8_0.

What does Q4_K_M mean in a file name?

It is an encoding label. The specification lists MOSTLY_Q4_K_M among its file types and the Hugging Face page describes Q4_K as 4-bit quantization at 4.5 bits per weight, but neither explains the M suffix in plain words, so check the model card.

Can Ollama run a GGUF file I downloaded?

Yes. Put FROM /path/to/file.gguf in a Modelfile and run ollama create. Ollama does not quantize during import, so the file must already be in the quantization you want.

What is a split GGUF file?

A model divided into numbered shards such as 00003-of-00009 in the file name. When you import one into Ollama, keep the original split file names and use a wildcard in the FROM line.

Sources

Sources checked November 3, 2026. Figures that depend on your exact model, runtime or machine are estimates; check the source or test on your own hardware.

Adults only (18+). Private uncensored AI · Terms · Privacy · All guides