AI glossary · Basic · Language models
Gradient descent
Gradient descent is a way for AI models to learn by adjusting their parameters little by little in the direction that reduces how wrong they are (the loss). Most deep learning models today are trained with this method or variations of it.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Models & generation
- Fields
- Language models
- Added
- 2026-10-08
How it works
First, the model makes a prediction with its current parameters, and a loss function measures how far the prediction is from the correct answer. Next, it computes how much and in which direction the loss changes when each parameter is changed slightly, that is, the gradient. In neural networks, this gradient is computed efficiently with backpropagation. The parameters are then adjusted slightly in the opposite direction of the gradient, where the loss decreases. This process is repeated countless times until the loss is small enough.
Why it matters
Gradient descent is the key tool that makes it practical to train huge neural networks. Without people defining rules one by one, a model can tune its own parameters given just data and a loss function. LLM pretraining and fine-tuning also ultimately run on this principle. That is why it is considered one of the first concepts to learn to understand deep learning.
Things to know
The value that sets how big each step is is called the learning rate. If the learning rate is too large, training overshoots the minimum and becomes unstable; if it is too small, training becomes too slow. In practice, stochastic gradient descent (SGD), which computes gradients on small batches (mini-batches) instead of the full dataset, and improved methods such as Adam are mainly used. There is also no guarantee of finding the point with the lowest loss, and training often stops at a point that is good enough.
Example
When training an LLM used in a chatbot, the model predicts the next token in a sentence and computes the loss by comparing it with the actual next token. It repeats adjusting the parameters to reduce that loss over enormous amounts of text, and this 'method of adjusting' is an optimization algorithm from the gradient descent family. The same principle is used when a company fine-tunes a model on its own data.