AI glossary · Basic · AI security · Agents · Data & evaluation
ExploitGym
ExploitGym is a benchmark that measures whether AI agents can turn already-known security vulnerabilities into actual attack effects. It was released as an arXiv paper in May 2026 and consists of 898 tasks drawn from real vulnerabilities.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Security · safety
- Fields
- AI security, Agents, Data & evaluation
- Added
- 2026-11-14
What it is
In security, a vulnerability is a weak spot in a program. On its own, it is not yet an attack. Real damage happens only when the weakness is used to go as far as reading files or executing code without permission, and this process is called exploitation. The paper's authors considered this capability very important but not properly evaluated so far, so they built ExploitGym. They aimed for an evaluation tool that is large-scale, diverse and close to real-world conditions.
How it works
In each task, the agent receives one program input that triggers a vulnerability. It must then expand this input step by step into a working exploit. The tasks come from real vulnerabilities in three areas: ordinary user programs, Google's V8 JavaScript engine and the Linux kernel. By changing the configuration of the same vulnerability, such as turning security protections on or off, you can also separately see how much defenses affect the success rate. Every configuration is packaged in a container (an isolated execution environment), so anyone can rerun it under the same conditions.
Why it matters
According to the paper, exploitation is still hard, but top models actually exploited a substantial number of vulnerabilities. The strongest configurations were Anthropic's Claude Mythos Preview and OpenAI's GPT-5.5, which produced working exploits on 157 and 120 tasks, respectively. Even with widely used defenses turned on, success rates were not negligible. Since then, as AI companies have released models specialized for cybersecurity, they have increasingly cited ExploitGym scores as evidence of performance.
Things to know
Exploitation capability is a dual-use technology. Defenders can use it to prioritize patches or verify risks, but it also lowers the barrier to entry for attackers. So the score is read not only as an indicator of 'how useful a model is' but also of 'how dangerous it could become.' Scores can also vary greatly depending on protection settings and task scope, so when comparing numbers from different sources, first check whether the conditions are the same. Some figures circulating in the community are unverified.
Example
On this site, ExploitGym was mentioned in news about OpenAI's cybersecurity model 'GPT-6 Cyber.' According to reports, OpenAI said that in an ExploitGym comparison GPT-6 Cyber scored 48.5% and the previous model, GPT-5.6 Cyber, scored 32.1%. The model was described as finding vulnerabilities, producing fixes, and being trained to operate only within authorized scope. Around the same time, a claim circulated in the community that an internal OpenAI model scored 77.0%, but this is unverified hearsay and the measurement conditions are unknown. When security teams look at such scores, it is best to first check which area and protection settings the numbers were measured under.
Where it came up on this site
- briefing Shared presentation material: "OpenAI's unreleased cybersecurity model scores 77.0% on ExploitGym… Claude Mythos 5.2 at 39.2%" (community hearsay, unverified) 2026-11-13
- briefing OpenAI releases cybersecurity model 'GPT-6 Cyber'… finds vulnerabilities, produces fixes, and is trained to operate only within authorized scope 2026-11-05