AI glossary · Basic · Efficiency & infra · Language models
GGUF
GGUF is a model file format used to run LLMs in llama.cpp-family tools. It packs information such as the tokenizer and chat template into a single file along with the weights, and mainly stores quantized weights.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Added
- 2026-10-12
How it works
A GGUF file contains both metadata (model architecture, tokenizer, chat template, etc.) and tensors (weights). So tools such as llama.cpp, Ollama and LM Studio can load the model and chat with it from this single file. Weights are usually stored quantized, reducing each number to fewer bits and greatly shrinking file size and memory use. A defining feature is that it is designed to run easily on CPUs, laptop GPUs and Macs.
What's inside one GGUF file
- model-Q4_K_M.gguf
- Metadata
- Model architecture
- Tokenizer
- Chat template
- Tensors (weights)Mostly quantized
Getting a feel for it
If you open a Hugging Face repository with 'GGUF' in its name, you will see several files with different names such as Q8_0, Q5_K_M and Q4_K_M. The number after Q is roughly the number of bits used per number; the smaller it is, the lighter the file, but quality drops a little at each step. A rule of thumb is 'number of parameters × bits ÷ 8 ≈ weight memory'. For example, a 7-billion-parameter model stored at 4 bits takes a little over about 3.5GB, and additional memory is needed for the KV cache for context.
Good to know
GGUF is mainly used in the llama.cpp family, so inference engines for GPU servers usually use safetensors or GPTQ/AWQ formats. When a new model architecture comes out, llama.cpp has to support it before GGUF conversion and execution are possible. Many files are converted and uploaded by the community rather than the official company, so it is a good idea to check the original model, its license and who did the conversion. Rather than always choosing the smallest quantization, a safe choice is to pick the highest bit count your memory allows.
Example
This site posted news (community report) that JetBrains released a small MoE model in GGUF. In a case like this, users pick one file from the repository, such as Q4_K_M or Q8_0, that fits their laptop's memory, load it into Ollama or LM Studio, and start chatting with it locally right away.