AI glossary · Basic · AI security · Computer vision
Adversarial example
An adversarial example is an input that has been deliberately altered by a small amount so that it looks almost unchanged to a human but leads an AI model to a completely different judgment. Think of it as an 'optical illusion' built to fool the model.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Security & safety
- Fields
- AI security, Computer vision
- Added
- 2026-10-09
How it works
An AI model decides on its answer by relying on subtle patterns in the input. An attacker finds the direction of change to the input that shakes the model's judgment the most, then adds a very small change (noise) in that direction. For an image, this means changing pixel values slightly; for text, it means changing a single word or an inconspicuous character. To a person it is the same image or the same sentence, but the model gives an entirely different answer. When the attacker can look inside the model, it is called a white-box attack; when the attacker works it out only by sending queries and seeing the answers, it is called a black-box attack.
White-box attacks vs. black-box attacks
White-boxKnows the model's internals
- Directly calculates the direction that shakes the judgment
- Targets precisely with few attempts
Black-boxSees only queries and answers
- Observes responses while changing the input
- Closer to attacks on real services
공통 Both flip the model's judgment with small changes that people do not notice
Why it matters
In many areas, such as autonomous driving, facial recognition, and spam and malware detection, AI judgments are directly tied to safety and security. If a small manipulation can flip a judgment in these areas, it can lead to serious harm. Adversarial examples also show clearly that models rely on statistical patterns rather than understanding 'meaning' the way people do. For this reason, they are also used as a standard tool for testing a model's robustness.
Things to know
A common defense is 'adversarial training', in which adversarial examples are deliberately created and mixed into the training data. However, a model that becomes resistant to one type of attack often remains vulnerable to other kinds of attacks, so the general view is that no complete defense exists yet. In LLMs, jailbreak attacks that bypass safety measures by appending strange strings are sometimes explained by the same principle. Increasing robustness can come at a cost, such as a slight drop in everyday accuracy.
Example
A company running an image recognition service has a red team create adversarial examples before deployment to check how easily the model can be fooled. For example, it feeds a product photo review model images with noise added that is invisible to the human eye, checks whether the model lets prohibited products pass as normal ones, and feeds the results back into the training data.