AI glossary · Beginner · Language models · Efficiency & infra
Local LLM
A local LLM is a language model you download and run directly on your own PC, laptop or company server instead of using a cloud API. Tools such as llama.cpp and Ollama make this process easier.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Infrastructure & using models
- Added
- 2026-10-12
How it works
You download an open-weight model file and load it into your device's GPU or CPU memory, and an inference engine reads it to generate answers. llama.cpp is a lightweight inference engine that mainly reads quantized models in GGUF format so they can run on ordinary PCs. Ollama is a tool that bundles downloading models, running them and serving a local API into a single command. Apps can send requests to an address on your own computer and use it like a cloud API.
Layers of a local LLM
- App · chat interfaceRequest to a local address
- Runner toolOllama etc.: download · run · API
- Inference enginellama.cpp etc.
- Model fileGGUF quantized weights
- Your deviceGPU · CPU · memory
Getting a feel for it
The first thing to look at is memory. Weight memory is roughly 'number of parameters × bytes per number,' plus headroom for things like the KV cache. For example, quantizing a 7-billion-parameter (7B) model to 4 bits (about 0.5 bytes) brings the weights alone to a little over 3.5 GB. Labels like 7B and Q4 in a model's name, along with the GGUF file size, let you estimate whether it will fit in your VRAM or unified memory.
Things to know
Data does not leave your device and there are no per-call fees, but expecting the performance of large cloud models easily leads to disappointment. Heavier quantization reduces memory but can lower answer quality. Licenses differ by model, and some prohibit commercial use, so check them. Downloaded model files and tools are also subject to supply chain security, so their sources need to be verified.
Example
Teams that cannot send internal documents to external APIs because of security rules often use Ollama to run open-weight models on company servers or developer laptops. This site's news included an update that Ollama switched to using the MLX runtime by default on Apple Silicon, which shows that even the same model can feel faster or slower depending on whether the execution engine suits the device.