AI glossary · Basic · MLOps & LLMOps · Language models
LLMOps
LLMOps refers to the management practices and tools needed to build and operate services that use LLMs (large language models). It builds on MLOps and adds work that is new with LLMs, such as prompt management, answer quality evaluation and cost management.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Data, ops & platforms
- Fields
- MLOps & LLMOps, Language models
- Added
- 2026-10-08
How it works
In LLM services, changes are more often made around the model than to the model itself. Prompts and system prompts are managed by version, and if you use RAG, the documents to retrieve and the embedding pipeline are managed too. Every change is checked for answer quality with Evals (evaluation sets), and in production, requests and responses, token usage and response speed are logged and monitored (LLM observability). When using multiple models, calls are sometimes centralized and managed through an LLM gateway.
Why it matters
LLMs answer the same question slightly differently each time, and there is often no single correct answer, which makes quality checks tricky. Changing a single line of a prompt can break other answers, so a process for comparing before and after a change is needed. Costs also rise directly with usage, and risks such as hallucination and prompt injection need ongoing attention during operation. A prototype may be easy to build, but this kind of management is essential to grow it into a stable service.
MLOps and LLMOps
MLOpsOperating self-trained models
- Managing data and the training process
- Performance metrics compared against ground truth
- Monitoring data drift
LLMOpsOperating LLM-based services
- Versioning prompts and retrieval documents
- Evaluating answers with many valid responses
- Managing token costs, hallucination and injection
공통 For both, the core is iterative operation: monitoring and fixing even after deployment
Things to know
LLMOps extends MLOps rather than replacing it, so fundamentals such as data management, deployment and monitoring are still needed. Answer quality evaluation often mixes human review with LLM-as-a-Judge (having another LLM score the answers), but keep in mind that the scoring LLM can also be wrong. If you use an external API model, behavior can change when the provider updates the model, so it is good to recheck periodically with the evaluation set. The scope of the term varies somewhat from company to company.
Example
Picture a team running a customer service chatbot. Every time they edit the prompt, they first score the answers against an evaluation set of frequently asked questions and ship the change only if the score does not drop. In operation, they watch conversation logs and token usage on a dashboard and collect cases with off-target answers to add to the next evaluation set. This system of repeating 'change → evaluate → deploy → observe' is LLMOps.