AI glossary · Basic · AI security · Safety & alignment
Red teaming
Red teaming is a testing activity in which people deliberately take an attacker's perspective and try to trick and misuse an AI model in order to find its weaknesses.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Security & safety
- Fields
- AI security, Safety & alignment
- Added
- 2026-10-08
How it works
The name red team comes from the team that plays the enemy in military exercises. AI red teams attack models in many ways, such as attempting jailbreaks, trying to extract dangerous information, or steering the model toward biased answers. This can be done directly by human experts, or another AI model can generate large numbers of attack prompts for testing. The weaknesses found are used to retrain the model or strengthen guardrails.
Why it matters
Even the company that built an AI model finds it hard to know all of its behaviors in advance. Red teaming is nearly the only way to surface, before release, the problems that real users or malicious actors will run into. That is why major AI companies often compile and publish red teaming results before releasing a new model. For areas requiring specialized knowledge, such as cybersecurity and biological or chemical risks, outside experts are sometimes brought in separately for testing.
Things to keep in mind
Red teaming is an activity for finding weaknesses, not a guarantee that a model is safe. Attacks using methods that were not tested may still remain. The attack techniques used in red teaming can themselves be misused, so managing who gets access and at what level is also important. It is ongoing work that has to be redone every time the model changes.
Example
A report featured on this site said that Anthropic reorganized its cybersecurity-related access system into three tiers: defense, red team and special. This can be seen as a move to set apart red team use, where security experts test systems as attackers would, and to manage the model access needed for such activities.