AI Trend Research

← AI glossary

AI glossary · Basic · Language models

Loss function

A loss function is a rule that calculates, as a single number, how far an AI model's prediction is from the correct answer. Training is the process of gradually adjusting the model to make this number (the loss) as small as possible.

About this term
Level
Basic · Technical terms you often meet in AI news
Added
2026-10-08
In plain words It is like a ruler that measures how far a dart landed from the bullseye. The farther away, the worse the score, and the player adjusts their stance to reduce that distance. However, what counts as 'far' depends on how the loss function is defined.

How it works

When the model receives an input and makes a prediction, the loss function compares that prediction with the correct answer and assigns an error score. For problems that predict a number, such as house prices, a common approach is to square the differences between predictions and correct answers and average them (mean squared error). For problems that choose a category, such as cat vs. dog, or for an LLM predicting the next token, cross-entropy is widely used; it penalizes the model more heavily the lower the probability it gives the correct answer. Based on the resulting loss value, backpropagation calculates which direction each parameter should change, and gradient descent adjusts the parameters little by little. Repeating this countless times reduces the loss.

The training loop driven by the loss functionTraining data → Model; Model → Prediction; Prediction → Loss function; Correct answer → Loss function; Loss function → Parameter update(Loss value); Parameter update → Model(Repeat)Training dataInputModelCurrent parametersPredictionLoss functionPrediction vs. correctanswerCorrect answerParameter updateBackpropagation · gradient descentLoss valueRepeat
The training loop driven by the loss function The process repeats: compare the prediction with the correct answer to get the loss, adjust the parameters to reduce it, then predict again.

Why it matters

The loss function is the only standard that tells the model 'what doing well means.' The model does not directly understand human intent; it changes only in the direction that lowers this number. So even with the same data and model, defining a different loss function can produce a model with an entirely different character. Techniques such as fine-tuning, RLHF and DPO can also be seen as ultimately being designs for how to express a goal as a loss (or reward).

Things to know

A low loss does not always mean a good model. If the loss is very low only on the training data and high on new data, suspect overfitting. Also, if the loss function does not accurately capture the goal you actually want, the model may learn odd tricks that only lower the number. That is why, during training, the loss on separately held-out validation data and actual evaluation results are examined together.

Example

In LLM pre-training, the model is shown the beginning of a sentence and asked to predict the next token, and the cross-entropy loss measures how high a probability it gave to the actual next token. Model development teams watch this loss curve go down throughout training to confirm that training is going properly.

Related terms

Gradient descent Backpropagation Overfitting Supervised learning Pre-training Activation function