AI glossary · Basic · Language models
Activation function
An activation function is a small transformation rule that decides how much of the signal each neuron in an artificial neural network passes to the next layer, and in what shape. This function is what allows a neural network to learn complex patterns beyond simple straight-line relationships.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Models & generation
- Fields
- Language models
- Added
- 2026-10-08
How it works
A single neuron in a neural network multiplies each of several inputs by a weight (importance) and adds them all up. Instead of passing this sum on as is, it runs it through an activation function once and sends the result to the next layer. The widely used ReLU is a very simple rule that turns negative values into 0 and leaves positive values unchanged. Sigmoid squeezes any value into the range between 0 and 1, and tanh maps it to between -1 and 1. Recent large language models also often use functions such as GELU or SwiGLU, which are smoothed-out versions of ReLU.
Why it matters
Without activation functions, no matter how many layers you stack, the result is equivalent to a single simple multiply-and-add calculation. The network could then only solve problems that can be separated by a single straight line. Because activation functions insert "bends" along the way, neural networks can represent complex, curved relationships such as those in images and language. In other words, they are an essential component for deep learning to gain the benefit of "depth."
Things to keep in mind
The choice of activation function affects training speed and stability. Functions whose outputs flatten out at both ends, like sigmoid, tend to cause a problem where the learning signal almost disappears as layers get deeper (vanishing gradients). ReLU became widely used because it reduces this problem, but it can also cause some neurons to keep outputting only 0 and drop out of learning. For this reason, modified functions continue to be used and studied for different purposes.
Example
Inside the transformer, the architecture behind LLMs such as ChatGPT and Claude, each layer contains a small neural network (the feed-forward layer), and activation functions such as GELU or SwiGLU are used there. Image classification models attach sigmoid or softmax at the end to output results as probability values between 0 and 1, such as "the likelihood that this is a cat."