AI glossary · Basic · AI platforms & solutions · Efficiency & infra
Kubernetes
Kubernetes is an open-source management system that automatically spreads containers (a program and its runtime environment packed into one box) across multiple servers, restarts them when they stop, and scales them up or down to match usage. It is often abbreviated as K8s.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Added
- 2026-10-08
How it works
The user writes the 'desired state', such as "run this many copies of this program, and it needs a GPU", in a configuration file. Kubernetes's brain, the control plane (API server, scheduler, etc.), receives this and places pods (the smallest unit that runs containers) on nodes, the actual servers, choosing ones with suitable resources. Lightweight work such as a web API can go on CPU nodes, while heavy work such as model serving can go on GPU nodes. After that, it keeps monitoring the current state, and if it differs from the desired state, for example when a pod dies, it starts it again to bring things back in line.
Why it matters
When servers grow to dozens or hundreds, it becomes hard for people to deploy and manage programs on each one by hand. Kubernetes handles deployment, failure recovery and autoscaling in a standard way, and has become the de facto common language of cloud infrastructure. In AI, it is widely used as the foundation for running model serving, training jobs and data pipelines on GPU clusters and for letting multiple teams share expensive GPUs.
Things to know
It has many components and concepts, so it is not easy to learn or operate. That is why many companies use managed Kubernetes run for them by cloud providers (e.g., Amazon EKS, Google GKE, Azure AKS). Using GPUs requires additional setup such as drivers and device plugins, and even a small misconfiguration can keep training or serving from running properly. For small services, it may be more than is needed.
Example
The news covered on this site, 'NVIDIA releases AICR v1.0, a configuration validation tool for GPU Kubernetes clusters', shows how important it is to check that settings are correct when GPUs are pooled together with Kubernetes. In practice, many companies run their LLM serving servers on Kubernetes, automatically adding model serving pods when requests surge and scaling them down when traffic is light to save on GPU costs.