llama.cpp

Overview

llama.cpp is an open-source project that runs LLM and VLM inference in C/C++. You can install it on your computer, download models and run them right away.

Install it with the install script, Docker, prebuilt binaries or a source build. The llama cli command downloads GGUF models from Hugging Face for chatting, and the llama serve command starts an OpenAI-compatible API server with a built-in web UI.

It suits developers who want to run open models locally with little overhead or start their own API server to connect to apps.

What it is used for

• Download GGUF models from Hugging Face and chat in the terminal.
• Start an OpenAI-compatible API server and call the model from your app.
• Chat with local models through the built-in web UI.
• Run VLM sessions that take images as input.

Pricing and license

MIT license

Getting started

1. Install with curl -LsSf https://llama.app/install.sh | sh (on Windows, irm https://llama.app/install.ps1 | iex).
2. Download and run a model with llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF .
3. If you need an API server, run llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF .

Links

Official site ↗Repository ↗

Related terms and companies

Local LLM GGUF Inference engine VLM Hugging Face

Recent mentions in briefings

• Ollama v0.40.4 RC adds decision models and EmbeddingGemma 2, shows local models in Codex model picker 2026-10-14
• AWS publishes guide to deploying decision models on SageMaker AI with llama.cpp, using SemIf to return per-option probabilities 2026-10-13
• Precision Numerics releases Boole-1.7B, a 247MB open-weight 1-bit quantization of Qwen3-1.7B (per community post) 2026-10-13

Sources

• ggml-org/llama.cpp README

Compiled from public sources (official pages, READMEs). Pricing and features change, so check the official site.